mikaell commented 4 years ago

Handling Hundreds of Files in Housekeeper

ping @dnil, @moonso, @henrikstranneheim, @jemten, @patrikgrenfeldt

The Three-body problem

Chromograph integration is stuck in the intersection of CG, Housekeeper and Scout. The reason is not knowing how the many (hundreds?) picture files should be handled and also to generate a configuration file pointing to the correct files. This is an invitation for discussion. Here are suggested solutions to the problem.

An ideal solution should:

keep track of files + delete files
not create a bloated view in Housekeeper
be conceptually true to existing software
sustainable in the long run
swiftly implemented

Tar-ball (current)

The current solution is to create a tar-ball and hand it to Housekeeper. This solution is not working at the moment. Problems arise when unpacking it is implossible to know the filenames and therefore generate a configuration to Scout.

A solution is to guess filesnames with hearuistics and a strict naming convention.

(+)

(-) Who handles the tracking and deleting of the unpacked files? Not working at the moment

Do Nothing

Add hundreds of picture files as is into existing software.

(+) No implementation Conceptually true to Housekeeper

(-) The view from housekeeper get will be bloated

Squash Housekeepers View

Tweak Housekeeper's default view by squashing series of filenames -example below. Verbose flag would show everything.

$ housekeeper get soundsnake
[...]   
/home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_[1-22].png 
[...]

$ housekeeper get soundsnake -V
[...]
384502 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_10.png |
384503 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_11.png |
384504 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_12.png |
384505 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_13.png |
384506 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_14.png |
384507 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_15.png |
384508 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_16.png |
384509 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_17.png |
384510 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_18.png |
384511 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_19.png |
384512 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_1.png |
[...]

(+) Conceptually true to Housekeeper Housekeepers view is not bloated Small implementation

(-)

Add Directory Handling in Housekeeper

Add functionality to Housekeeper to add and handle directories (adding dirs is already possible, but not delete them).

(+) Small implementation Housekeepers view is not bloated

(-) Changes Housekeeper concept Have not thought through semantics of this change

Merge PNGs Into One

Instead of handling hundreds of picture files, merge them into one.

(+) Housekeepers view is not bloated

(-) Loose granularity and flexibility Not future proof for adding functionality

Rewrite Chromograph in Javascript

Rewrite picture generation to be done in Scout just in time.

(+) ?

(-) Unknown implementation time Unknown problems

Other solutions are welcome! Preferably this can be decided quickly.

mikaell commented 4 years ago

I vote for squashing the default view.

dnil commented 4 years ago

I'd say squashing is a good option, regardless of how our three-body problem decomposes? Did you read the SF book by the way, or are you referring to the original physics issue? 😸

dnil commented 4 years ago

Rewrite to js lacks immediacy, and can always be done later without interfering with anything else, as long as the track data is anyway passed to Scout, so leave that one off for now.

Merge is still somewhat tempting to me, but it may be slightly slower to implement, is indeed somewhat more rigid as well as a little less immediate in conveying information to the user.

moonso commented 4 years ago

With regards to #71 we need to skynda långsamt whatever solution is chosen

mikaell commented 4 years ago

Did you read the SF book by the way, or are you referring to the original physics issue? 😸

Both I suppose. Haven't got around reading the book yet. I limit myself to one thick book per year.

mikaell commented 4 years ago

New Tags For Housekeeper

Type	Old Tags	Suggested New Tags
upd (regions)	ACC5322A2, chromograph	ACC5322A2, chromograph, upd, regions
upd (site)	ACC5322A2, chromograph	ACC5322A2, chromograph, upd, sites
Tcov	ACC5322A2, chromograph	ACC5322A2, chromograph, tcov

-replace ACC5322A2 with sample_id.

jemten commented 4 years ago

Do we need a chromosome tag as well or are you planning to get that from the filename later?

mikaell commented 4 years ago

Do we need a chromosome tag as well or are you planning to get that from the filename later?

Ideograms will be placed in public/static and committed into Scout.

jemten commented 4 years ago

Ideograms will be placed in public/static and committed into Scout.

Right! Then, I'll drop MIP's functionality to generate ideograms

seallard commented 4 months ago

Stale

Clinical-Genomics / housekeeper