TeaDrinkingProgrammer commented 1 year ago

Describe what problem your feature request solves A Flatpak install would be a great alternative to Snaps and Appimages, since this allows most users to download threat-dragon straight from their software store. Would people be open for this? I could look into it myself but I want to see if there is an appetite for it.

Flatpack instructions and flathub along with electron and flatpak

jgadsden commented 1 year ago

This would be great to have @TeaDrinkingProgrammer , if you would like to find out what is involved then that would be great

sparticvs commented 1 year ago

@TeaDrinkingProgrammer I'm interested in this too. That is where I'd like to see this go for easier installation on Fedora and to bypass the Fedora package management process. It is something that I was looking at on v1.6, but I haven't had a moment to look at it for v2. If you are interested, but don't know where to start, I can take this, but I have prioritized getting the German translation committed.

jgadsden commented 1 year ago

We probably could follow the example in https://docs.flatpak.org/en/latest/electron.html it takes several steps, but nothing that can not be automated I have assigned this to both of you @sparticvs and @TeaDrinkingProgrammer , hope that is OK and you can collaborate on this

TeaDrinkingProgrammer commented 1 year ago

@sparticvs @jgadsden I was still planning on looking at it, it is just that I am incredibly busy right now, I probably will have some time in a week or two. I have never done anything with Flatpak though, so it will be a learning experience for me. How about I start working on it and ask for help/feedback when I have something to show for?

jgadsden commented 1 year ago

Yes, sure, that sounds great @TeaDrinkingProgrammer - over to you

TeaDrinkingProgrammer commented 1 year ago

I did a bit of research and bumped into this fun roadblock: The guide @jgadsden linked is indeed very useful and I followed it. This went well until I got to the step where you have to build the NPM packages. This step uses a separate python script to download all NPM dependencies so you do not have to put them in the manifest manually. The problem is that this script relies on a package-lock file from either NPM or YARN to get the download links. I have looked up if maybe it would be possible to convert from PNPM lock files to NPM lock files, but the creator of PNPM has the staunch opinion developers should add support for PNPM instead of PNPM adding a convertor (I think this is quite unrealistic to expect from users and that it will put off developers rather than help adoption, but I digress).

I think there are a few ways of solving this issue:

Switching to NPM: I do not know the ins and outs of PNPM vs NPM, but I imagine it was chosen for a reason.
Adding PNPM support to the script: this would be possible, the script is not very big, but I personally wouldn't want to undertake it as I am not very fluent in Python, Flatpak or PNPM.
Only using NPM for generating a package-lock: I have tried this out. Besides the fact that the only-allow package would need to be disabled for this, it also creates a package-lock with different versions than the main one, which will probably create some headaches.

I honestly do not like any of the options, but I do not see any others. Any ideas?

jgadsden commented 1 year ago

Hello @TeaDrinkingProgrammer - I realise the amount of work to get this far, much appreciated

pnpm was chosen because it is fast on the download and install ... and we liked using it as it is marginally more forgiving than npm. But that was the only reason: we thought it looked fun and went with it

It has caused us some problems, for example moving from pnpm version 7 to 8, and it does have some downsides

If it is stopping us from doing flatpak then we can migrate back to npm, it is not as bad as it sounds and does simplify our pipeline scripts

We can ponder for a bit? and if we decide to move back to npm then I am happy to implement this, unless you want the honour :)

jgadsden commented 1 year ago

I was intending to do a 2.1 release soon, this will reinstate the PDF button on the desktop version

I could delay this for a bit so as to get npm back in if we decide that is the way forward ... all part of the pondering :)

TeaDrinkingProgrammer commented 1 year ago

@jgadsden No, I'm happy to hand it back over to you, it is a lot more complicated than I thought 😅 , but hey, I learned a lot from it. Probably, moving to NPM is the obvious option as it is the industry default, but the script also supports YARN so that would also be an option. From what I can find, Yarn is faster than NPM, but slower or equal to PNPM, but I find that these kinds of benchmarks or opinions are rather subjective. One interesting thing about Yarn, is that it does not have an audit fix command (you need a package for this), I can imagine this could be a dealbreaker for an app under the OWASP umbrella. I would personally give them both a try and see what works, but that's just me.

jgadsden commented 1 year ago

oops, I did not mean to say that I wanted to implement the flatpak, I was just referring to the migration back to npm from pnpm I hope you will still carry on with the flatpak @TeaDrinkingProgrammer ?

sparticvs commented 1 year ago

I don't think we should migrate back to NPM from PNPM just for flatpak. We can convert from Appimage into flatpak with relative ease, but IMO it's more about getting the pipeline fully assembled and tested before merging it into the upstream (aka here). I tried doing a very thorough approach with the flatpak on v1.6, but the issues I've been running into have more to do with electron (I was attempting to do a full native flatpak setup, instead of a smarter way).

@jgadsden I would assume that we would want to have some "official" Flathub account for the flatpaks to go into. Not sure if this is something OWASP already has or if a new account would need to be created.

@TeaDrinkingProgrammer if you haven't yet, do some investigation on turning an AppImage into a flatpak. IIRC, AppImage is one of the current build artifacts and IIUC conversion should be fairly straight forward. If I was following your commentary on the research, it sounded like you were trying to get the running of PNPM in the flatpak as a way to keep the image up-to-date with the release pipeline. While I think that is a good goal, let's just solve that problem with a pipeline run for now. Nightly updates aren't unheard of, and I can volunteer some personal infrastructure to ensuring that happens/kicks off.

jgadsden commented 1 year ago

@sparticvs I have added your comment to the discussion #636 'npm or pnpm?', I hope that is OK with you?

sparticvs commented 1 year ago

A follow up to my comment earlier - I went looking and I'm not finding exactly what I was reading before about AppImage -> Flatpak conversion. There is a comment in a list somewhere Alexander Larsson (flatpak creator) that mentions that there are a bit more differences that make this conversion difficult, so I remember incorrectly and wanted to correct that before @TeaDrinkingProgrammer went down a rabbit hole.

TeaDrinkingProgrammer commented 1 year ago

I have heard about Fedora turning RPMs into flatpaks, would that be worth looking in to?

sparticvs commented 1 year ago

@TeaDrinkingProgrammer yes, although I haven't done that myself, I know others that have.

jgadsden commented 1 year ago

We have moved Threat Dragon back to npm from pnpm. There is no real difference, npm install times now seem to be comparable with pnpm and we are no longer forcing our developers to use pnpm. It is still OK to use pnpm with Threat Dragon, we do not get in the way of anyone installing and running with pnpm, it is just that now the repo clones with a package-lock.json instead of apnpm-lock.json file

@TeaDrinkingProgrammer did you want to carry on with your development? you probably just need to sync your fork to get the npm lock file

TeaDrinkingProgrammer commented 1 year ago

It's been a while, but I haven't forgotten about this. I have looked at it again with NPM and I have gotten a lot further, but I am up to a roadblock again. I have a strong hunch it has to do with the Python script but I will discuss it here first to rule out any oddities on this side.

The problem is as follows:

As I discussed previously, Flatpak does not allow internet connections in the build step. Flatpak does support downloading sources before the build step, so there is a script which "converts" the package-lock.json file to a generated-sources.json file. This allows flatpak to download all node_modules dependencies before the buildstep which are then used by NPM during npm install. I generate the generated-sources file first and let Flatpack build, but get this error:

npm verb cli /usr/lib/sdk/node18/bin/node /usr/lib/sdk/node18/bin/npm
npm info using npm@9.5.1
npm info using node@v18.16.1
npm verb title npm install
npm verb argv "install" "--offline" "--loglevel" "verbose"
npm verb logfile logs-max:10 dir:/run/build/threat-dragon/flatpak-node/npm-cache/_logs/2023-07-03T10_26_11_573Z-
npm verb logfile /run/build/threat-dragon/flatpak-node/npm-cache/_logs/2023-07-03T10_26_11_573Z-debug-0.log
npm verb stack Error: request to https://registry.npmjs.org/lru-cache/-/lru-cache-9.0.0.tgz failed: cache mode is 'only-if-cached' but no cached response is available.
npm verb stack     at cacheFetch (/usr/lib/sdk/node18/lib/node_modules/npm/node_modules/make-fetch-happen/lib/cache/index.js:12:13)
npm verb stack     at async fetch (/usr/lib/sdk/node18/lib/node_modules/npm/node_modules/make-fetch-happen/lib/fetch.js:98:7)
npm verb cwd /run/build/threat-dragon/main
npm verb Linux 6.3.6-200.fc38.x86_64
npm verb node v18.16.1
npm verb npm  v9.5.1
npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/lru-cache/-/lru-cache-9.0.0.tgz failed: cache mode is 'only-if-cached' but no cached response is available.
npm verb exit 1
npm verb unfinished npm timer reify 1688379971595
npm verb unfinished npm timer reify:unpack 1688379971684
npm verb unfinished npm timer reifyNode:node_modules/path-scurry/node_modules/lru-cache 1688379971685
npm verb unfinished npm timer reifyNode:node_modules/is-core-module 1688379971686
npm verb code 1

This error suggests that lru-cache-9.0.0 has not been cached before the build step. This is seems logical, since the package_lock.json specifies ^9.0.0 and this the generated_sources gets the latest version that specifies that, which is 9.0.1. What confuses me is why NPM does not understand this when installing in offline mode.

Files:

generated-sources-and-flatpak.zip

Commands used:

flatpak-node-generator npm -r package-lock.json flatpak-builder build flatpak/org.owasp.threat-dragon.yml --install --force-clean --user

Update: I have experimented with Yarn just to see if that would help, but Yarn v3 is not supported yet

jgadsden commented 1 year ago

I will try these commands on my Linux system, and see how far I get

mademarc commented 11 months ago

Hey @jgadsden can i contribute to this issue? and if so how can i help & contribute to get tis resolve ASAP.

Please and thanks :)

jgadsden commented 11 months ago

Hello @mademarc , if you would like to repeat what @TeaDrinkingProgrammer has done so far, that would be a great first step

jgadsden commented 3 weeks ago

@sparticvs , @TeaDrinkingProgrammer and @mademarc , is the flatpak / flathub support progressing? We moved to npm and package-lock.json (instead of a pnpm-lock.json file) so it may be worth revisiting No problem if we can not do Flatpak in time for version 2.3 (end August 2024) as it can be bumped back to version 2.4 (early December 2024) or version 3.0 Q1/Q2 2025

TeaDrinkingProgrammer commented 3 weeks ago

I might look into it in a couple of weeks so if I find the time, it would be in time for 2.3.

jgadsden commented 3 weeks ago

Thanks @TeaDrinkingProgrammer If I understand correctly, there is no need for a flathub account? The submission is done following page Submission which does not need a specific account

mademarc commented 3 weeks ago

hey @jgadsden i was unable to address the Flatpak feature request asi was super busy at that moment and that said i could consider the following steps to assist that are below:

Research Flatpak Packaging: Look into creating a Flatpak package, especially for Electron apps.

2.Test Locally: Build and test the Flatpak package on their system.

Submit to Flathub: Follow Flathub’s submission guidelines to make it available.
Update Documentation: Add Flatpak installation instructions to the project's documentation.

or can i follow up with @TeaDrinkingProgrammer in order to fix this issue ?

jgadsden commented 3 weeks ago

you are most welcome to work on this @mademarc , I do not have the time myself but if you could coordinate with @TeaDrinkingProgrammer that would be great

OWASP / threat-dragon

Flatpak/flathub support #626

The problem is as follows:

Files:

Commands used: