search

IDR / idr-metadata

Curated metadata for all studies published in the Image Data Resource
https://idr.openmicroscopy.org
14 stars 24 forks source link

idr0051-fulton-tailbudlightsheet S-BIAD815 #653

Open dominikl opened 1 year ago

dominikl commented 1 year ago

Export time: 17 min / plate Import time: 4 hours.

dominikl commented 1 year ago

Currently converting on pilot-zarr1-dev , will be available in /data/idr0051.

will-moore commented 1 year ago

Looks done:

(base) [wmoore@pilot-zarr1-dev ~]$ ls -l /data/idr0051
total 0
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 12:18 180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 12:53 180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.ome.zarr
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 14:36 2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.ome.zarr
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 14:46 embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.ome.zarr
drwxrwxr-x. 4 dlindner dlindner 76 Jun 12 15:40 embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.ome.zarr
dominikl commented 1 year ago

👍 Yes, finised.

will-moore commented 1 year ago

To zip each dir... (remove / etc)

for i in */; do zip -r "${i%/}.zip" "$i"; done

will-moore commented 1 year ago 
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 mb s3://idr0051
make_bucket: idr0051
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-policy --bucket idr0051 --policy file://policy.json
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-cors --bucket idr0051 --cors-configuration file://cors.json

Should have run in a Screen as this took too long and lost connection...

/home/wmoore/mc cp -r idr0051/ uk1s3/idr0051/zarr
...g_preview.klb.ome.zarr/0/1/45/0/338/0/0: 14.32 GiB / 14.43 GiB ━━━━━━━
will-moore commented 1 year ago

Started running again, in a screen this time...

/home/wmoore/mc cp -r idr0051/ uk1s3/idr0051/zarr

...
$ /home/wmoore/mc cp -r idr0051/ uk1s3/idr0051/zarr
...e.zarr/OME/METADATA.ome.xml: 18.03 GiB / 18.03 GiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.92 MiB/s 
$ /home/wmoore/mc ls uk1s3/idr0051/zarr
[2023-06-26 13:10:54 UTC]     0B 180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr/
[2023-06-26 13:10:54 UTC]     0B 180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.ome.zarr/
[2023-06-26 13:10:54 UTC]     0B 2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.ome.zarr/
[2023-06-26 13:10:54 UTC]     0B embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.ome.zarr/
[2023-06-26 13:10:54 UTC]     0B embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.ome.zarr/
will-moore commented 1 year ago

Initial rendering in vizarr is poor because single channel is grey:

Image

since vizarr doesn't take into account the greyscale setting.

Seen in validator: Image

will-moore commented 1 year ago 
$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0051/idr0051 bsaspera_w@hx-fasp-1.ebi.ac.uk:5f/136e8d-xxxxxxxxxxx
180712_H2B_22ss_Courtney1_20180712-163837_p00                                                                         100% 1818MB 84.0Mb/s    02:02    
180712_H2B_22ss_Courtney_p00_c00_reg_preview.                                                                         100% 5430MB  210Mb/s    07:48    
2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c                                                                         100%  878MB 87.3Mb/s    08:44    
embryo_dmso_2_new_17-00-44_p00_c00_reg_previe                                                                         100% 2156MB  205Mb/s    11:15    
embryo_dmso_2_new_17-00-44_p00_c00_reg_previe                                                                         100% 2153MB 95.9Mb/s    13:46    
Completed: 12736265K bytes transferred in 827 seconds
 (126115K bits/sec), in 5 files, 1 directory.
will-moore commented 11 months ago

Following link from https://www.ebi.ac.uk/biostudies/submissions/ I can open the submission at https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD815

I was hoping that the files for that submission would be available at a URL: https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD815.html but this gives:

<Error>
  <Code>NoSuchKey</Code>
  <BucketName>bia-integrator-data</BucketName>
  <RequestId>tx000000000000001633b49-0064d1ba0d-18ba7307-default</RequestId>
  <HostId>18ba7307-default-default</HostId>
</Error>

(this URL is based on previous idr0054 study data being available at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD704.html

will-moore commented 11 months ago

Data available at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD815.html E.g. https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97/51afff7c-eed4-44b4-95c7-1437d8807b97.zarr/0

To harvest the uuids we need...

let csv = "";
$("#viewable tbody tr").each(function() {
    let $this = $(this);
    if ($("a", $this).length == 0) return
    let uid = $( "a:first", $this).attr("href").replace(".html", "");
    let zarrzip = $( "td:nth-child(3)", $this).text();
    csv += `${zarrzip},${uid}\n`
});
console.log(csv);

Gives me:

idr0051/180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.ome.zarr.zip,S-BIAD815/51afff7c-eed4-44b4-95c7-1437d8807b97
idr0051/embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.ome.zarr.zip,S-BIAD815/b2633930-86b0-489e-a845-d2a7afe6ff15
idr0051/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.ome.zarr.zip,S-BIAD815/c49efcfd-e767-4ae5-adbf-299cafd92120
idr0051/2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.ome.zarr.zip,S-BIAD815/e12a8e2a-4fce-4579-a78b-b0c4597c3ada

Only 4 out of 5 zips are "viewable" above (unzipped onto s3). idr0051/embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.ome.zarr.zip failed to unzip

will-moore commented 10 months ago

Moving back to EBI column since we don't yet have all 5 images on s3 yet. Discussed via e-mail. EBI will look at getting all images processed.

will-moore commented 10 months ago

This has been tested with mkngff at https://github.com/IDR/omero-mkngff/pull/4#issuecomment-1683943256.

will-moore commented 7 months ago 
(venv3) [wmoore@test120-omeroreadonly-1 scripts]$ python check_pixels.py Project:552 --max-planes=sizeC
Start: 2023-12-06 20:45:32.863680
Checking Project:552
max_planes: sizeC
max_images: 0
0/5 Check Image:4007817 180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.pattern
1/5 Check Image:4007818 180712_H2B_22ss_Courtney_p00_c00_reg_preview.klb.pattern
2/5 Check Image:4007819 2018-06-28_21ss_DMSO_TF_20180628-185945_p00_c00_reg_preview.pattern
3/5 Check Image:4007820 embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.pattern
4/5 Check Image:4007821 embryo_dmso_2_new_17-00-44_p00_c00_reg_preview.klb.pattern
End: 2023-12-06 20:45:49.510165