We just did a new build, and I noticed some things that seemed a little off. I don't think they're major, but worth looking into at some point.
Sub-tasks
Some of these are issues, some are questions, and some are ideas.
[ ] 1. Fix: Certain things appearing to not run when they are supposed to
[ ] metadata/*metrics.json changes, but none for docs/metrics/ (i)
It seems to me that the latter should always happen if there are updates to the former. Bug?
[ ] migrate-SOURCE.md changes, but none for slurp/SOURCE.tsv (i)
It seems to me that the former should always happen if there are updates to the latter. Bug?
[ ] 2. Fix: Spurious diffs (sorting)
Todo: We may be getting spurious diffs on a lot of files due to lack of record sorting. We should find if that's the case and fix.
migrate*.md and unmapped*.md: I think this is might be causing large diffs in these files as well (i).
lexmatch/README.md: Needs sorting (#177)
[ ] 3. Analyses
[ ] 2 builds back-to-back
I think we should do a test and run a build back-to-back on the same branch. We should hope to see that there is no diff between the the results. I suppose we could do this either of 2 ways: (a) run the build, commit, run again, or (b) run on the same commit twice. Hopefully, in the time that these builds are running, no sources being wgeted will have been updated, otherwise that will sully the results.
[ ] Build logs
I noticed some lexmatch errors that are probably worth not ignoring. I realize now that I have not usually been checking build logs. It may be that there are other errors and warnings popping up that we should not be ignoring, indicative of problems needing to be fixed.
[ ] 4. Error controlling
E.g. strict vs non. We may want to consider in some cases throwing errors rather than printing errors or warnings to the console. For example, see the lexmatch error above (build logs). Another error we've recently turned into a warning is an 'ancestor' issue in slurp. We could potentially introduce an environmental variable for controlling whether or not certain errors should halt execution, and we could set a default in the makefile. However, we'd need to pass this as an argument into our Python scripts.
[ ] 5. Specific errors
These are only happening sometimes. Having difficulty replicating.
However, I wonder if some of these issues might be caused by how I handled merge conflicts. I was going to merge develop into main (#572), but I got some conflicts. They were all data outputs, and since I'd just run a build on develop, I considered develop to be authoritative, and so I used that version of the file in all cases. These were the conflicts:
```
src/ontology/external/nando-mappings.robot.owl
src/ontology/external/nord.robot.owl
src/ontology/external/ordo-subsets.robot.owl
src/ontology/imports/omo_import.owl
src/ontology/imports/ro_import.owl
src/ontology/lexmatch/README.md
src/ontology/metadata/doid-metrics.json
src/ontology/metadata/gard-metrics.json
src/ontology/metadata/icd10cm-metrics.json
src/ontology/metadata/icd10who-metrics.json
src/ontology/metadata/icd11foundation-metrics.json
src/ontology/metadata/ncit-metrics.json
src/ontology/metadata/omim-metrics.json
src/ontology/metadata/ordo-metrics.json
src/ontology/reports/component_signature-omim.tsv
src/ontology/reports/mirror_signature-omim.tsv
src/ontology/reports/omim_mapping_status.tsv
src/ontology/reports/omim_unmapped_terms.tsv
src/ontology/slurp/omim.tsv
src/ontology/unmapped/omim-unmapped.owl
```
Overview
We just did a new build, and I noticed some things that seemed a little off. I don't think they're major, but worth looking into at some point.
Sub-tasks
Some of these are issues, some are questions, and some are ideas.
metadata/*metrics.jsonchanges, but none fordocs/metrics/(i)migrate-SOURCE.mdchanges, but none forslurp/SOURCE.tsv(i)migrate*.mdandunmapped*.md: I think this is might be causing large diffs in these files as well (i).lexmatch/README.md: Needs sorting (#177)wgeted will have been updated, otherwise that will sully the results.strictvs non. We may want to consider in some cases throwing errors rather than printing errors or warnings to the console. For example, see the lexmatch error above (build logs). Another error we've recently turned into a warning is an 'ancestor' issue in slurp. We could potentially introduce an environmental variable for controlling whether or not certain errors should halt execution, and we could set a default in the makefile. However, we'd need to pass this as an argument into our Python scripts.525
Additional information
We did a build on June 17th:
571
572
573
Merge conflicts play any role?
However, I wonder if some of these issues might be caused by how I handled merge conflicts. I was going to merge develop into main (#572), but I got some conflicts. They were all data outputs, and since I'd just run a build on develop, I considered develop to be authoritative, and so I used that version of the file in all cases. These were the conflicts: ``` src/ontology/external/nando-mappings.robot.owl src/ontology/external/nord.robot.owl src/ontology/external/ordo-subsets.robot.owl src/ontology/imports/omo_import.owl src/ontology/imports/ro_import.owl src/ontology/lexmatch/README.md src/ontology/metadata/doid-metrics.json src/ontology/metadata/gard-metrics.json src/ontology/metadata/icd10cm-metrics.json src/ontology/metadata/icd10who-metrics.json src/ontology/metadata/icd11foundation-metrics.json src/ontology/metadata/ncit-metrics.json src/ontology/metadata/omim-metrics.json src/ontology/metadata/ordo-metrics.json src/ontology/reports/component_signature-omim.tsv src/ontology/reports/mirror_signature-omim.tsv src/ontology/reports/omim_mapping_status.tsv src/ontology/reports/omim_unmapped_terms.tsv src/ontology/slurp/omim.tsv src/ontology/unmapped/omim-unmapped.owl ```
Related