Open seandavi opened 2 months ago
In #19 I have code to download and unzip all of the listed DRC "dcc" and "file" assets (the "code" assets don't really have a concrete thing to download). Unzipped, here is the file breakdown:
(2) .1 files
(2) .2 files
(2) .3 files
(721) . files
(350) .csv files
(19) .zip files
(8) .download files
(1,422) .txt files
(264) .json files
(11,206) .tsv files
(47) .gmt files
(19) .tsv~ files
(4) .pdf files
(2) .docx files
(2) .swp files
(8) .sh files
(2) .obo files
(2) .gz files
(6) .DS_Store files
(2) .3-cfde-submission_74d840d264792e9d7ab38fcb0deeba0eddbbebdb files
(166) .numbers files
(2) .json_orig files
(2) .tsv_error files
(2) .tsv_orig files
(12) .orig files
(2) .uri files
(8) .new files
14,284 files
124.4 GB
Also, between "dcc" and "file", they are mostly overlapping. There's about 7k entries in each, there are only ~600 that are in one but not the other. The rest are exact matches.
The file contents are things like node/edge lists, compounds, samples, genes, and all sorts of stuff. Could you give me some guidance on what high-level info information I should compile from this that would be useful for program evaluation? I'm not sure what would be useful here.
Also, the data is naturally too big to store in the repo here for providence. I have the ingest process download all this data to /raw/temp
, which is excluded with gitignore.
Could you give me some guidence
Thanks, @vincerubinetti, for the details. I do not foresee a reason for downloading all the data. I need to investigate the contents of the DRC download link files to see what we can and should be tracking.
I have it all downloaded on my laptop, which took a long time, so if you want to take a look at anything in particular or collect some high-level info, let me know.
Personally, looking through all of it, I don't see anything there that would necessarily be useful for the purposes of evaluating the effectiveness of certain projects. In fact, I didn't see anything to link a particular asset to a particular project. But maybe a few of them have it and I just missed it.
Perhaps if we could associate assets to projects, things like "last updated" and "number of genes/nodes/edges" could be useful? Though even that feels like measuring programmer effectiveness via lines of code written (not accurate).
Nonetheless, #20 adds the infrastructure to do things like downloading large/many data files at once, unzipping them, etc... which we may want to do later anyway.
This issue just captures the information shared by the DRC with us via email to help@cfde.cloud. These links were a response to my asking for API access to the information on https://info.cfde.cloud and https://data.cfde.cloud.
Currently, each file has at most a couple hundred rows. I think these correspond to links in this Data Matrix View
current dcc assets format
current code assets format
current file assets format