Repo cleanup post-changes

[manulera]

Hello @kimrutherford and @ValWood.

I have made a bunch of changes in the repository, following some help from the obo people.

I think we no longer need to run the release procedure locally, and we can do it using github actions.

There are two separate actions:

• refresh imports:
  • Updates the imports into a single file merged_import instead of using all the separate files that was using below.
  • Creates a pull request that we then have to accept to merge into master.
  • This should be done before the other action, or when new terms from an ontology, after commiting the new terms to the .txt files (e.g. chebi_terms.txt). It takes ~40 mins to run.
• run release:
  • Runs the release pipeline and creates a draft release with all the release artifacts uploaded. See this draft release that I created yesterday.

The release action does not commit anything to the repository, and that is because other ontologies do not keep those release artifacts in source control, only the equivalent to fypo-edit. We have already moved fypo.owl and others out of source control because they became too big, so I think it would be safe to remove all the other release artifacts as well, and keep them only as release attachments. However, we have to think if us or someone else is using these files from the repo url and not the release url. We already ammended some urls when we moved fypo.owl out, is there anything else that comes to mind?

In addition, I will take the occasion to remove some files that are no longer needed, e.g. all the .owl and .obo files from the imports folder except for merged_import.owl.

cc: @matentzn does this make sense? Anything else we could clean up since we are doing it?

[manulera]

@matentzn what about the files in this folder https://github.com/pombase/fypo/tree/master/imports should a merged_import.owl be committed there as well? These were updated with the former release pipeline, but the update imports from now seems to only affect this file: https://github.com/pombase/fypo/blob/master/src/ontology/imports/merged_import.owl

Then, we used to get reports in reports/fypo-edit.owl-obo-report.tsv and src/ontology/reports/fypo-edit.owl-obo-report.tsv (identical). What should we do about those?

[ValWood]

Brillaint, thanks for doing this, and thanks @matentzn and @gouttegd for the help!

[kimrutherford]

Great work! That's a big improvement.

[manulera]

Hi @kimrutherford @matentzn.

• Kim: Do we use urls of ontologies in the repo that would be moved to the release attachments?
• Nico: Any thoughts on who should be warned about the changes in the structure of the repo? Someone that does not uses purl obolibrary urls?

[manulera]

PS: I will include the warnings as a release file.

[matentzn]

@manulera I think it is sufficient to add a prominent warning on the release notes on GitHub.

Remind people to.

Never ever ever ....

use URLs to refer to data and ontology files on the web.

Always use PURLs and only PURLs!

[kimrutherford]

We have already moved fypo.owl and others out of source control because they became too big, so I think it would be safe to remove all the other release artifacts as well, and keep them only as release attachments

Hi @manulera

Could you explain what you mean by "release attachments". I don't understand.

Kim: Do we use urls of ontologies in the repo that would be moved to the release attachments?

We use these OBO files, which we read directory from a local checkout of the fypo repository:

• supplemental_files/fypo_extension_relations.obo
• fypo-simple-pombase.obo
• fyeco.obo

Never ever ever .... use URLs to refer to data and ontology files on the web.

We definitely don't do that. :100:

Always use PURLs and only PURLs!

Oh dear, we don't do that either. :frowning_face:

[matentzn]

Local checkout is also not reliable, because file locations change within a repo.. if you use purls and curl the ontology from the PURL, than you can be certain that you will always grab the correct file

[manulera]

Hi @matentzn, thanks for your responses. There is one more thing I wanted to ask about. I ran the refresh imports and nothing changed (expected), however this is added to the diff:

https://github.com/pombase/fypo/pull/4331/files

I am not sure what this field is used for. Will it be a problem if we make a release on a different date, say 2023-05-16. Does this mean we always have to run it the same day?

[matentzn]

The release and imports date are very different things! Dont worry, this is exactly as it should be. For the import, the date denotes when the import was last refreshed.

[ValWood]

Hi, I tried to run the new release pipeline for the first time today.

I got an error on refresh imports https://github.com/pombase/fypo/actions/runs/5201493460/jobs/9381726725#step:7:237

@matentzn if you have any info about the error codes please let me know, (@manulera is on vacation so I am stuck until he returns).

I also posted a question on Slack.

Thanks if you can advise.

[ValWood]

From Nico: the problem is that you have not enough memory set tun run the merged import, see https://oboacademy.github.io/obook/howto/odk-setup/#problems-with-memory-important

[matentzn]

Sorry I should have read this better. You did not run on your machine, you ran in GH actions.. See slack, this is a much bigger problem, and I am not sure if it can be solved at all.. is @manulera not there?

[ValWood]

He is in Japan...

[ValWood]

Manu has run it successfully though....

[matentzn]

Yea he may have been lucky though. He must have been close to the memory limit..

[kimrutherford]

I've committed a change which seems to have helped: https://github.com/pombase/fypo/commit/61e36f45f43dc9c919ded475751518c6117eb7e7

The "Refresh imports" Action finished after that: https://github.com/pombase/fypo/actions/runs/5204542287

[manulera]

Bravooo @kimrutherford

[ValWood]

excellent, thanks Kim

[ValWood]

The second part of the pipeline ran OK.