PNG and SVG download per pathway

[AlexanderPico]

Do we want to support the download of individual PNG and SVG images per pathway page? We have the data.wikipathways.org site for monthly downloads. But download options per pathway page are different.

1. We could have SVG and PNG files in the wikipathways-database/pathways/WPID/ folders. But would this unnecessarily bloat the repo that may be forked by all contributors?
2. We could have a separate repo, e.g., wikipathways-images that hold these and maybe other "downstream" files that don't otherwise need to be in the main repo.
3. Other ideas?

[AlexanderPico]

There are download options on the new site per pathway page already, but they don't work. They are just placeholders until we figure this out.

[khanspers]

I think it would be nice to support direct download options on each page. Option 2 seems to make the most sense. In that case, an update to a gpml in wikipathways-database/pathways/WPID/ would have to trigger an update of the relevant svg/png etc in wikipathways-images, right?

[ariutta]

I like idea 2. But if one of the download formats is GMT, we might want to pick a different name than wikipathways-images.

[ariutta]

@khanspers, the triggering of the update would be pretty straightforward. We would create the PNG and SVG images like this, but instead of adding them to wikipathways-database, we'd add them to a different repo, somewhat similar to how we do it for the sync-site-repo jobs.

[AlexanderPico]

Cool. We wouldn't have a GMT download for a single pathway, but indeed we might want more than just images in this accessory repo. So, how about wikipathways-assets for all accessory files downstream of the GPML that aren't needed in the wp-db repo.

• PVJSON
• SVG
• PNG (converted from SVG)

Maybe the info.json file too? And the .md and .tsv files should really only be in the website repo. These aren't needed in the wp-db repo at all, right? Move instead of copy? Separating these downstream files in their own repo would reduce the number of bot commits/pushes to the main wp-db repo, which might be smarter and safer as well.

[AlexanderPico]

https://www.npmjs.com/package/convert-svg-to-png

[AlexanderPico]

Attempted to update yml to generate future json and svg file in new wikipathways-downstream repo: https://github.com/wikipathways/wikipathways-database/blob/main/.github/workflows/on_gpml_change.yml#L276

[andrawaag]

Would it be possible to create direct download links, instead of (or next to) a separate repo? With my experiments on Wikimedia commons where I upload timestamped versions of pathways, my current workflow is to download either the SVG or PNG version to my desktop. From there I upload them to commons and manually add the metadata.

Ideal would be if it would be possible to either add PNG or SVG to the pathway link (e.g. https://www.wikipathways.org/index.php/Pathway:WPxx.png). Uploading to Commons would no longer require an intermediate download step, which would make uploading/update to commons scale. This would also facilitate the automatic inclusion of metadata in the process.

It would also all direct inclusion in for example HTML pages.

[AlexanderPico]

@andrawaag Yes! With GH Pages set on the repo, URLs like this work: https://new.wikipathways.org/wikipathways-assets/pathways/WP554/WP554.svg or https://assets.wikipathways.org/pathways/WP554/WP554.svg

[AlexanderPico]

@ariutta I've got most of the updated json-svg job working, but have an error within the gpmlconverter script: line 246: gpml2pvjson: command not found

See error in this run: https://github.com/wikipathways/wikipathways-database/runs/6243515696?check_suite_focus=true

The NPM install appears fine, but apparently this job can't access it for some reason. I'm stumped.

[AlexanderPico]

The issue is that I'm installing npm in the scripts/generate-svgs subdir, but I'm running the script from the parent dir. I'm doing this because the paths in the GH Action VM are very confusing, but this is not going to work... Need to figure out paths, I guess.

[AlexanderPico]

Defined ${{ github.workspace }} paths at $AS_PATH and $DB_PATH. That appears to have worked, but now I'm getting this odd error: getopt: command not found. But getopt is part of util-linux which is already installed and up-to-date. Sigh...

https://github.com/wikipathways/wikipathways-database/runs/6283056934?check_suite_focus=true

[AlexanderPico]

Ok. Fixed the $PATH variable (my mistake) and now the basic functions are found and working. Now I just have to figure out how to get a deploy key for the assets repo...

ERROR: Permission to wikipathways/wikipathways-assets.git denied to deploy key fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

[AlexanderPico]

Added ACTIONS_ASSETS_KEY to wp-db repo and wikipathways-database assets key to wp-assets repo, but that didn't work. Checkout even failed! Restoring ACTIONS_DEPLOY_KEY restored successful checkout, but the commit and push fail with the same error as above.

[AlexanderPico]

Ah, I think I need that weird Install SSH Client step! It has an odd comment that made me think it was temp code, but it makes sense that it's needed for ssh-based checkout and push. Will test and then updated the comment if needed.

[AlexanderPico]

It works! Key steps to dealing with a secondary repo within a job:

• Generate a new key pair on your local machine, give it a unique name to not confuse with other keys
• Add public key as a deploy key to secondary repo (the one being pushed to), and check Allow Write box
• Add private key as a secret (including comment lines) to primary repo (the one hosting the action)
• Add Install SSH Client to job after checkout

[AlexanderPico]

Reopening for:

• [x] Testing generated SVGs (e.g., linkouts and metabolite-uris??)
• [x] Impl svg-to-png. Try https://www.npmjs.com/package/convert-svg-to-png

[AlexanderPico]

Update: ID unification on gene product datanodes is not working in pvjson generation, thus SVG linkouts are failing. See #65 for possible solution.

[AlexanderPico]

Fixed linkouts with suggestion from @ariutta to insert quotes into wd query cmd.

[AlexanderPico]

Large and thumb png from svg working too. Populated wp-assets repo and set on_gpml_change.yml to update all future changes.

[AlexanderPico]

This should be the official url pattern: https://assets.wikipathways.org/pathways/WP554/WP554.svg

But this will also work: https://new.wikipathways.org/wikipathways-assets/pathways/WP554/WP554.svg