Open jwodder opened 1 month ago
Thank you for this and #83 -- I love idea of storing more metadata about test runs and underlying assets/blobs properties. I think I lean more toward this over #83
status.yaml
could "duplicate" outcome
field to mitigate.status
untested
and add a record for an asset when new asset detected but was not scheduled to be tested yet in the run etc. Then we could also deduce how long it took from new asset detection to actually getting it testedstatus.yaml
files then could be just "reproduced" from records
at any point. dandiset_id
would be helpful{dandiset_id}/.dandi/records/{year}.jsonl
sounds good to me since would allow for easier cleaninggit-annex forget
(and may be git replace
grafting to do keep some archive with full history).environment
: If we are to contain all such good provenance, I wonder if we should also get top level singular environments.jsonl
where we would create records (env_id
identified by a isodate stamp when change in setup detected) providing more detailed description of the system, beyond only versions of top level modules (output of uname -a
for kernel info, versions of numpy
, h5py
etc) which we collect upon initiation of a run, compare to last one and possibly record a new one; and then referring to that env_id
in the test record run (instead of versions of top level modules). This would allow to notice changes in setup, identify test runs for the same setup, etc. I believe this is somewhat reminiscent on what asv
does in its machine.json
or alike.timestamp
is more of datetimestamp
, i.e. would be fully fledged iso datetime stamp, right?@yarikoptic
I think I lean more toward this over https://github.com/dandi/dandisets-healthstatus/issues/83
So should #83 be closed as "not planned"?
May be just
status.yaml
could "duplicate"outcome
field to mitigate.
If you'd like that, please file a dedicated issue.
what if to add
status
untested
and add a record for an asset when new asset detected but was not scheduled to be tested yet in the run etc.
Currently, if dandisets-healthstatus
detects an asset that exists on the Archive but not in status.yaml
, the asset won't be added to status.yaml
unless it's selected for testing. Thus, if we emit a record for new assets, the same records would be emitted every time the program is run until the respective assets are finally tested.
With code separation (as discussed in #72) keeping only summary/extract statuses in a repo sounds odd but would allow anyone to quickly clone it, so I still think that we might want to keep such a detailed log and outputs (#72) in a separate repo which we might "forget" etc, and just link it as a submodule.
To be clear, here you're just agreeing with me that we should have one repository that stores the code and a separate repository that stores all of the status.yaml
files, issue 72 output records, and issue 82 event logs, correct? (However, I think making the logs+records repository a submodule of the code repository — if that's the direction you're suggesting — isn't that great of an idea, as the code isn't dependent on the logs, and updating the code repository whenever there's a commit to the logs repo would just introduce a lot of churn.)
I wonder if we should also get top level singular
environments.jsonl
... and then referring to thatenv_id
in the test record run
Would this env_id
also be used to replace the versions
mappings in status.yaml
?
in above I assume
timestamp
is more ofdatetimestamp
, i.e. would be fully fledged iso datetime stamp, right?
Yes.
@yarikoptic Ping.
May be just
status.yaml
could "duplicate"outcome
field to mitigate.If you'd like that, please file a dedicated issue.
https://github.com/dandi/dandisets-healthstatus/issues/86
what if to add
status
untested
and add a record for an asset when new asset detected but was not scheduled to be tested yet in the run etc.Currently, if
dandisets-healthstatus
detects an asset that exists on the Archive but not instatus.yaml
, the asset won't be added tostatus.yaml
unless it's selected for testing. Thus, if we emit a record for new assets, the same records would be emitted every time the program is run until the respective assets are finally tested.
I see we have already some untested
but it is not test specific (e.g. here). We can add records with status untested
(relates to #86) and provide untested
per each test as soon as we encounter new asset.
With code separation (as discussed in #72) keeping only summary/extract statuses in a repo sounds odd but would allow anyone to quickly clone it, so I still think that we might want to keep such a detailed log and outputs (#72) in a separate repo which we might "forget" etc, and just link it as a submodule.
To be clear, here you're just agreeing with me that we should have one repository that stores the code and a separate repository that stores all of the
status.yaml
files, issue 72 output records, and issue 82 event logs, correct?
I meant to have separate repo only for (heavy) logs (this #82) as a submodule of (lean) repo status.yaml
files. If you would like to have code/
to be a separate repo (and submodule) - could also be separated out.
(However, I think making the logs+records repository a submodule of the code repository — if that's the direction you're suggesting — isn't that great of an idea, as the code isn't dependent on the logs, and updating the code repository whenever there's a commit to the logs repo would just introduce a lot of churn.)
I had in mind making logs/
to be the submodule of this dandisets-healthstatus
repo containing status.yaml
files. code/
also could be submodule if so desired, not the other way around (no logs submodule within code).
I wonder if we should also get top level singular
environments.jsonl
... and then referring to thatenv_id
in the test record runWould this
env_id
also be used to replace theversions
mappings instatus.yaml
?
yes, I think it could, hopefully environments
would not change too often thus complicating things up... may be environment id
could be composition of versions + date or alike, e.g.
20240725-092554_hdmf-3.14.2_matnwb-2.6.0.2_pynwb-2.8.1
so this way we could then immediately see what versions of the main libraries of interest were while the actual record would provide more gory details.
@yarikoptic
I meant to have separate repo only for (heavy) logs (this https://github.com/dandi/dandisets-healthstatus/issues/82) as a submodule of (lean) repo
status.yaml
files.
Do you mean that the event logs should be in a separate submodule from that used for the issue 72 output logs? Which one gets the logs/
name?
logs/
folder but it should be a submodule pointing to dandisets-healthstatus-logs
which would carry actual logs.@yarikoptic By "event logs," I mean the JSON Lines files storing a summary of each test run as described by this issue. Where do you want those files to be stored?
@yarikoptic Ping.
ok, what about this repo to keep records.jsonl
by the status.yaml
(as rendered from records.jsonl
). Then outputs/
to be submodule which would collect stdout/err for running each test as corresponding to the records in records.jsonl
in this repo?
@yarikoptic
You stated above that you preferred dividing the event logs up by year, so should records.jsonl
in your last comment be replaced by records/{year}.jsonl
?
I don't really like keeping the status.yaml
and event records in this repository when we have another one for the output logs, but OK.
- You stated above that you preferred dividing the event logs up by year, so should
records.jsonl
in your last comment be replaced byrecords/{year}.jsonl
?
sounds good to me if we are to do per year.
Proposal: Make the
check
andtest-files
subcommands ofdandiset-healthstatus
append one summary line per test run to one or more record file(s). Unlike thestatus.yaml
files, entries are never overwritten or removed.The record file(s) will be stored in the same repository used for #72. Possible paths for the file(s) include:
records.jsonl
— one file for everything{dandiset_id}/.dandi/records.jsonl
— one record file per Dandiset (placed under.dandi/
so as not to collide with any assets){dandiset_id}/.dandi/records/{year}.jsonl
— one record file per Dandiset and yearThe record file(s) will be in JSON Lines format, with each line being a JSON object containing the following fields:
asset
(object) — details on the asset the test was run on; fields:dandiset_id
(string)dandiset_version
(string) — For now, this will always be"draft"
path
(string) — the asset pathsize
(integer) — the asset's blob sizeasset_id
(string)blob_id
(string)modified
(timestamp) — the asset'smodified
property from the Archiveblob_modified
(timestamp) — the asset'sblobDateModified
metadata fieldtest
(string) — the name of the test ("pynwb_open_load_ns"
or"matnwb_nwbRead"
)timestamp
(timestamp) — starting time of the test processduration
(float) — duration of the test process in secondsoutcome
(string) —"pass"
,"fail"
, or"timeout"
environment
— an object listing the versions of software used by the tests, the same as theversions
mappings currently present in thestatus.yaml
filesenvironment
could also include adandisets-healthstatus
field whose value equals the current (short?) Git commit hash of the codeThis will allow looking back through records of past test runs (e.g., to compare past and present runtimes) without having to muck about with Git history.
(Compare #83, which proposes storing the above information in the
status.yaml
files.)@yarikoptic: Your thoughts?
Additional items added based on discussion below:
status.yaml
under each test with status "untested" (or "new"?).environment
above and theversions
field instatus.yaml
withenv_id
, an ID into a separate JSON (or JSON Lines?) fileuname -a
, numpy version, and h5py version.20240725-092554_hdmf-3.14.2_matnwb-2.6.0.2_pynwb-2.8.1