search

galaxyproject / galaxy_codex

Galaxy Communities Dock aka Galaxy Codex: catalog of Galaxy resources (tools, training, workflows)
https://galaxyproject.github.io/galaxy_codex/
MIT License
7 stars 12 forks source link

Collect error rate data for tools #134

Open abretaud opened 1 week ago

abretaud commented 1 week ago

It would nice to identify tools that are probably bugged: we can (somehow) look into the job exit codes on different usegalaxy.* servers, and tag tools with a "to fix" label when we see most of the jobs are failing.

I guess a tool can fail on a specific server and work fine on another one, not sure how to handle that (only tag when it fails everywhere? warn specific server admin where it fails?)

I recall @neoformit had some magic sql queries to identify problematic tools, maybe it could be used somehow?

neoformit commented 1 week ago

We did work on these scripts that enumerate jobs states and perform some crude clustering in an attempt to enumerate each tool-error. Maybe you can find them useful:

https://github.com/usegalaxy-au/tools-au/tree/master/scripts/broken-tools (check the README)

After some discussion at last GCC it seemed like the Galaxy Sentry server should be taken advantage of for this purpose. It seemed like modifying the sentry handler a little could provide most of the functionality above.

paulzierep commented 1 week ago

Thanks for the hint @neoformit that looks very interesting !!