Write document describing how to create a build manifest from the database

[magnusmanske]

First attempt here.

[podpearson]

Thanks @magnusmanske , I think this is very useful, though is somewhat different to what I was expecting. Comments below.

I've decided to reopen this as I think we should only close once we have a signed-off version (I'll close #15 now).

• [ ] I think before doing further work on this, we first need to be explicit about the scope of what is going to be in the first release. As such I think this isssue is dependent on completion of #6 and #7, so you should first complete those before coming back to this.
• [x] The introduction should have a brief description of what a manifest is and why we need to create these.
• [x] Describe the types of manifest that we might want to create, and be clear about which is in scope for first release and which is for later. Examples include:
  • All files for a given set of samples
  • All files for a given taxon
  • All files for a given study or set of studies
• [x] Is the long-term expectation that end-users will need to write SQL as described in this document? Or is the intention that there will be some method for creating manifests that doesn't require writing SQL, e.g. a command line such as fits_create_manifest --sample_ids_file <sample_ids_filename> or a web interface? I think I was expecting that the first release would have some way of creating manifests that didn't involve writing complex SQL queries. I think its fine if this isn't in scope for the first release, but I think we do need to state what the long-term goal is. If you already have ideas about what the long-term goal is, write them in here. If this still needs to be decided, have some statement here about the fact that this needs to be decided, and create a new issue to capture this that is referenced from here.
• [ ] Some of this document feels more like a description of the database (e.g. explanation of how tags work) rather than a description of how to make a manifest. Could you consider whether some of the text in here could/should be moved to the database description document (#9), and reference that document from here?
• [ ] I think the most important long-term use case is creating a manifest of all files for a given set of samples. Could you include this use case in here?
• [x] Could you also think about how the various pieces of documentation you are writing are going to fit together? E.g. one option would be to add a README to the repo (https://github.com/wtsi-team112/fits/new/master?readme=1) that has a brief description and links to all the documentation. Once you decided on how you'd like to link the documents together, it might be good to create an issue to track doing this.

[podpearson]

regarding Duplicate of qc_complete; not sure what this is supposed to be - description is not a duplicate of qc_complete, it is mlwh.iseq_run_status_dict.description (which should contain "qc complete" for all samples). Not vital at this stage though.

[magnusmanske]

Now here

[podpearson]

• [x] Please could you turn my previous list of comments into a set of checkboxes here, so we can keep track of what still needs to be done?

[magnusmanske]

@podpearson "feels like a description of the database" - I think the difference here is that the descriptions and commands in the document are specifically designed for manifest creation, whereas the database documentation gives more of a general overview.

"manifest for a given set of samples" That's in there, see "Likewise, to get all FITS samples that have one of your favourite Oxford codes (tag_id=3561)". You'll have to combine that with the manifest generation code below; hence, "This tutorial assumes a basic knowledge of MySQL syntax."

[podpearson]

"manifest for a given set of samples" That's in there, see "Likewise, to get all FITS samples that have one of your favourite Oxford codes (tag_id=3561)". You'll have to combine that with the manifest generation code below; hence, "This tutorial assumes a basic knowledge of MySQL syntax."

My reading of this is that this gives you all the FITS samples for one single Oxford code, but the more common use case I think is getting all data a list of Oxford codes.

[magnusmanske]

A list of all Oxford codes can be retrieved via SELECT DISTINCTvalueFROM vw_sample_tag WHERE tag_id=3561. You can, of course, specify this further. "This tutorial assumes a basic knowledge of MySQL syntax."

[podpearson]

@magnusmanske - I've just made a pull request with two suggested minor typos.

"There can only ever be one row for a tag/(sample/file)/value combination". Ids 3689178 and 7426287 in sample2tag both have the same values of sample_id (190), tag_id (3604) and value (1195-PF-TRAC2-DONDORP). Is this a mistake?

This issue should be left open until we have sign-off, i.e. agreement at a production meeting that this is good to go.

[podpearson]

Regarding the "Complex whitelist/blacklist filter" in the where clause, can you remind me exactly what these lists were? Are they essentially the diff between what taxon mlwh thinks files are, and what species we have them according to Solaris study_group? If so, I think a cleaner way to address this would be to have the taxon tag populated with information from Solaris study_group where this exists, but from mlwh taxon otherwise. Has this already been done?