search

IDR / idr-metadata

Curated metadata for all studies published in the Image Data Resource
https://idr.openmicroscopy.org
14 stars 24 forks source link

idr0015-colin-taraoceans S-BIAD861 #645

Open will-moore opened 1 year ago

will-moore commented 1 year ago

idr0015-colin-taraoceans

pwalczysko commented 1 year ago

Reimport still in progress - cancelled once because of long wait on FILESET_UPLOAD_PREP. The new import in progress since 8 March, also FILESET_UPLOAD_PREP (with parallel-upload=10)

jburel commented 1 year ago

Since we will be working on that study. We should take the opportunity to also fix the location metadata

pwalczysko commented 1 year ago

Imported without chunks and exchanged the symlink in ManagedRepo similarly to the idr0013 case. The new Plate on pilot-idrtesting is http://localhost:1080/webclient/?show=plate-254 and the name is idr0015-nochunks. All looks good, the thumbs and full viewer work fine.

will-moore commented 1 year ago

Estimate data volume...

uint8, 4 channels, Z: 20, 2048 x 2048, 22 x 18 wells, 84 plates. 11 TB

will-moore commented 1 year ago

Starting to free-up some space...

(base) [wmoore@pilot-zarr1-dev data]$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  4.4T  587G  89% /data

$ cd /data
$ sudo rm -rf idr0011/

$ df -h ./
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  3.6T  1.3T  74% /data
will-moore commented 1 year ago

Convert 1 screen...

screen -S idr0015_ngff
/home/wmoore/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.screen TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr

EDIT: permission denied - chown -R wmoore idr0015 then re-ran at 13:16...

will-moore commented 1 year ago

Make bucket...

$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 mb s3://idr0015
make_bucket: idr0015
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-policy --bucket idr0015 --policy file://policy.json
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-cors --bucket idr0015 --cors-configuration file://cors.json
will-moore commented 1 year ago

Upload a previously-created plate from pilot-zarr1-dev

cd /data/idr0015
$ /home/wmoore/mc cp -r TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr uk1s3/idr0015/zarr
.../V/9/0/3/0/4/9/0/0: 53.04 GiB / 53.04 GiB ━━━━━━━━━━━━━━━ 25.02 MiB/s 36m10s

Looks good in validator

https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0015/zarr/TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr

And vizarr (although the omero rendering settings in the .zarr are different from what's in IDR). E.g. only 3 channels active, although 5 are exported OK:

Screenshot 2023-07-02 at 07 35 34
will-moore commented 1 year ago

Uploaded a 2nd plate, recently generated above:

(base) [wmoore@pilot-zarr1-dev idr0015]$ /home/wmoore/mc cp -r TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr uk1s3/idr0015/zarr
...me.zarr/V/9/0/3/0/4/9/0/0: 79.03 GiB / 79.03 GiB ━━━━━━━━━━━━━━━━━━━━ 37.09 MiB/s 36m21
will-moore commented 1 year ago

Converting 4 more plates...

for i in TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U01--V01 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U00--V01 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V0; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw --memo-directory ../memo /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen ${i%.*}.ome.zarr; done
will-moore commented 1 year ago

Seeing errors writing memo files...

2023-07-02 20:38:56,486 [main] WARN  loci.formats.Memoizer - failed to save memo file: ../memo/uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/.TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01.screen.bfmemo
java.io.IOException: No such file or directory
will-moore commented 1 year ago

And then... (caused by a typo in my command above: --V0 should be --V00.

Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@63a65a25): java.io.FileNotFoundException: /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V0.screen (No such file or directory)
will-moore commented 1 year ago

Try to avoid memo issues by not using memo directory, but allow writing to source dir...

$ cd /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/
$ sudo chown wmoore patterns/

$ df -h /data/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  3.6T  1.4T  74% /data

Running 12 more (including repeat of last typo fix above):

for i in TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V01 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U01--V01 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U00--V01 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U01--V01 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U00--V01 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U01--V01 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U00--V01 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U01--V01 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U00--V01 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U01--V01 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U00--V01; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen ${i%.*}.ome.zarr; done
will-moore commented 1 year ago

Started to zip some... First 4 zarrs...

screen -S idr0015_zip
for i in TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U01--V01 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U00--V01 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V0; do zip -r "${i%/}.ome.zarr.zip" "$i.ome.zarr"; done
will-moore commented 1 year ago

Still seeing memo issues...

2023-07-03 18:41:39,091 [pool-1-thread-4] WARN  loci.formats.Memoizer - skipping memo: directory not writeable - /uod/idr/filesets/idr0015-UNKNOWN-taraoceans/20150918-tara/RAW_DATA/TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14/slide--S00/chamber--U00--V01/field--X17--Y21
will-moore commented 1 year ago

Zip command above created 3 zips but failed with same typo as earlier! (oops again):

zip warning: name not matched: TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V0.ome.zarr

Conversion above was generating memo errors as before. Last was:

2023-07-03 21:47:48,790 [pool-1-thread-4] WARN  loci.formats.Memoizer - skipping memo: directory not writeable - /uod/idr/filesets/idr0015-UNKNOWN-taraoceans/20150918-tara/RAW_DATA/TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10/slide--S00/chamber--U00--V01/field--X17--Y21

but ran to completion OK...

Current status...

$ ls -alh /data/idr0015
total 161G
drwxrwxr-x. 19 wmoore dlindner 4.0K Jul  4 05:08 .
drwxrwxr-x. 14 root   idr-data  270 Jul  2 22:53 ..
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  2 13:17 TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  2 18:23 TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    48G Jul  3 15:20 TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  2 20:03 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U00--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    63G Jul  3 22:49 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  2 21:33 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    51G Jul  4 05:08 TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 06:27 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore dlindner 4.0K Mar  9 11:30 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore dlindner 115K Mar  9 10:13 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.screen
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 07:54 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 09:33 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 10:44 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 11:55 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 13:03 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 14:14 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 15:24 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 16:29 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 18:42 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U00--V01.ome.zarr
drwxrwxr-x. 16 wmoore wmoore    232 Jul  3 20:34 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 21:47 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U00--V01.ome.zarr

Available space getting low...

$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  4.7T  234G  96% /data

Need to delete data... $ sudo rm -rf idr0036

Upload first 3 zips...

$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0015/idr0015 bsaspera_w@hx-fasp-1.ebi.ac.uk:5f/13xxxxxxxx
will-moore commented 1 year ago

Deleted 3 zips uploaded above and their .zarr dirs.

Move all remaining .zarr to batch1 for zipping and upload...

mkdir batch1
mv *.zarr batch1
cd batch1
for i in */; do zip -r "${i%/}.zip" "$i"; done
will-moore commented 1 year ago

Converting 10 Filesets into "batch2"...

screen -r idr0015_ngff
mkdir batch2
for i in TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U01--V01 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U00--V01 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U01--V01 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U00--V01 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U01--V01 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U00--V01 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U01--V01 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U01--V01 TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U00--V01; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen batch2/${i%.*}.ome.zarr; done
will-moore commented 1 year ago

Deleting individual ome.zarr filesets from batch1 once their ome.zarr.zip has been created.

Also upload a random (last) ome.zarr to s3 from batch1 to validate we're still good...

$ /home/wmoore/mc cp -r TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U00--V01.ome.zarr uk1s3/idr0015/zarr
...me.zarr/A/1/0/0/0/0/5/0/0: 7.45 MiB
...me.zarr/V/9/0/3/0/4/9/0/0: 60.48 GiB / 60.48 GiB ━━━━━━━━━━━━━━━━━━━━ 28.41 MiB/s 36m19s
will-moore commented 1 year ago

Current state of batch1: In a day of generating zips, we only have 4 zips created (5th due soon): Zips take 4-5 hours to generate, compared to only about 1.5 hours to convert the ome.zarr.

(base) [wmoore@pilot-zarr1-dev ~]$ ls -alh /data/idr0015/batch1
total 196G
drwxrwxr-x. 14 wmoore wmoore   4.0K Jul  6 20:21 .
drwxrwxr-x.  4 wmoore dlindner  138 Jul  5 22:55 ..
-rw-rw-r--.  1 wmoore wmoore    49G Jul  6 04:39 TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--.  1 wmoore wmoore    32G Jul  6 08:48 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x.  8 wmoore dlindner  107 Jul  6 23:09 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    31G Jul  6 13:12 TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 07:54 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    61G Jul  6 20:14 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 09:33 TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 10:44 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 11:55 TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 13:03 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 14:14 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 15:24 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 16:29 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 18:42 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U00--V01.ome.zarr
drwxrwxr-x. 16 wmoore wmoore    232 Jul  3 20:34 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  3 21:47 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U00--V01.ome.zarr
-rw-------.  1 wmoore wmoore    25G Jul  6 23:10 zitPNp7I

Current state of batch2 (started generating zips too):

(base) [wmoore@pilot-zarr1-dev ~]$ ls -alh /data/idr0015/batch2
total 88G
drwxrwxr-x. 12 wmoore wmoore   4.0K Jul  6 22:37 .
drwxrwxr-x.  4 wmoore dlindner  138 Jul  5 22:55 ..
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 00:29 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    38G Jul  6 17:24 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 01:41 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U00--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    45G Jul  6 22:29 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 02:53 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 04:06 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 05:11 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 06:27 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 07:45 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 08:59 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 10:16 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul  6 11:37 TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U00--V01.ome.zarr
-rw-------.  1 wmoore wmoore   5.1G Jul  6 23:14 ziBWfTRP
will-moore commented 1 year ago

Convert another 10 in batch3...

for i in TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U01--V01 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U00--V01 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U01--V01 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U00--V01 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U01--V01 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U00--V01 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U01--V01 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U01--V01 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen batch3/${i%.*}.ome.zarr; done
will-moore commented 1 year ago

Upload completed zips from batch1...

(base) [wmoore@pilot-zarr1-dev batch1]$ ls /data/idr0015/batch1/idr0015/
TARA_HCS1_H5_G100001472_G100001473--2013_09_28_19_45_25_chamber--U00--V01.ome.zarr.zip
TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U00--V01.ome.zarr.zip
TARA_HCS1_H5_G100002411_G100002481--2013_08_28_14_46_59_chamber--U01--V01.ome.zarr.zip
TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U00--V01.ome.zarr.zip
TARA_HCS1_H5_G100002479_G100002163--2013_08_26_14_59_40_chamber--U01--V01.ome.zarr.zip
TARA_HCS1_H5_G100002567_G100002568--2013_09_25_12_55_58_chamber--U00--V01.ome.zarr.zip
$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0015/batch1/idr0015 bsaspera_w@hx-fasp-1.ebi.ac.uk:5f/136xxxxxxxxxx
TARA_HCS1_H5_G100001472_G100001473--2013_09_2             100%   48GB  328Mb/s    23:53    
TARA_HCS1_H5_G100002411_G100002481--2013_08                  100%   31GB  364Mb/s    38:41    
TARA_HCS1_H5_G100002411_G100002481--2013_08_2.             100%   30GB  290Mb/s    53:57    
TARA_HCS1_H5_G100002479_G100002163--2013_08_2             100%   60GB  118Mb/s  1:32:42    
TARA_HCS1_H5_G100002479_G100002163--2013_08_2             100%   60GB  304Mb/s  2:03:27    
TARA_HCS1_H5_G100002567_G100002568--2013_09_2             100%   33GB 91.7Mb/s  2:20:28
will-moore commented 1 year ago

Zipping ran out of space...

  adding: TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U00--V01.ome.zarr/A/1/0/0/0/3/4/1/ (stored 0%)
  adding: TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U00--V01.ome.zarr/A/1/0/0/0/3/4/1/0
zip I/O error: No space left on device
zip error: Output file write failure (write error on zip file)
will-moore commented 1 year ago

For batch3 try zipping with -0 (no compression) and -m (move files into zip - delete originals)...

$ screen -r idr0015_zip
$ cd /data/idr0015/batch3
$ for i in */; do zip -0mr "${i%/}.zip" "$i"; done
will-moore commented 1 year ago

Upload 5 more from batch2, then delete

$ ls -alh /data/idr0015/batch2/idr0015
total 196G
drwxrwxr-x. 2 wmoore wmoore 4.0K Jul  7 14:58 .
drwxrwxr-x. 8 wmoore wmoore 4.0K Jul  8 00:04 ..
-rw-rw-r--. 1 wmoore wmoore  38G Jul  6 17:24 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  45G Jul  6 22:29 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  43G Jul  7 03:30 TARA_HCS1_H5_G100003584_G100003586--2014_06_26_15_58_43_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  37G Jul  7 08:14 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  34G Jul  7 12:17 TARA_HCS1_H5_G100003741_G100003739--2013_09_30_14_59_10_chamber--U01--V01.ome.zarr.zip

$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0015/batch2/idr0015 bsaspera_w@hx-fasp-1.ebi.ac.uk:5f/136e8d-xxxx
TARA_HCS1_H5_G100003584_G100003586--2013_09_2                                                                                       100%   37GB  313Mb/s    16:46
TARA_HCS1_H5_G100003584_G100003586--2014_06_2                                                                                       100%   44GB  177Mb/s    38:08
TARA_HCS1_H5_G100003584_G100003586--2014_06_2                                                                                       100%   42GB  271Mb/s    58:21    
TARA_HCS1_H5_G100003741_G100003739--2013_09_3                                                                                       100%   36GB  291Mb/s  1:16:30    
TARA_HCS1_H5_G100003741_G100003739--2013_09_3                                                                                       100%   33GB 64.6Mb/s  1:32:49

$ rm -rf /data/idr0015/batch2/idr0015/*
will-moore commented 1 year ago

Looks like zipping of batch3 (with -m and -0) stopped prematurely for some reason:

screen -r idr0015_zip2
...
  adding: TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr/A/1/0/0/0/4/19/0/ (stored 0%)
  adding: TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr/A/1/0/0/0/4/19/0/0 (stored 0%)
  adding: TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr/A/1/0/0/0/4/19/0/1 (stored 0%)

Last 2 zips are smaller than expected. Probably need to regenerate

(base) [wmoore@pilot-zarr1-dev batch3]$ ls -alh
total 614G
drwxrwxr-x. 2 wmoore wmoore   4.0K Jul  9 04:32 .
drwxrwxr-x. 5 wmoore dlindner  156 Jul  7 08:03 ..
-rw-rw-r--. 1 wmoore wmoore    96G Jul  8 05:04 TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    80G Jul  8 09:04 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    79G Jul  8 12:29 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    72G Jul  8 16:00 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    76G Jul  8 19:06 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    75G Jul  8 22:08 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    74G Jul  9 01:30 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    59G Jul  9 04:09 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore   6.6G Jul  9 04:31 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    55M Jul  9 04:32 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr.zip

but looks like space didn't run out:

$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  4.5T  482G  91% /data
will-moore commented 1 year ago

Zip remaining zarrs in batch1 (and batch2)

screen -S idr0015_zip(2)
cd batch1(2)
for i in */; do zip -0mr "${i%/}.zip" "$i"; done
will-moore commented 1 year ago

Moved all zips for batch3 and upload (except for 2 zips that appear too small - corrupted?).

(base) [wmoore@pilot-zarr1-dev batch3]$ ls -alh idr0015/
total 608G
-rw-rw-r--. 1 wmoore wmoore  96G Jul  8 05:04 TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  80G Jul  8 09:04 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  79G Jul  8 12:29 TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  72G Jul  8 16:00 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  76G Jul  8 19:06 TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  75G Jul  8 22:08 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  74G Jul  9 01:30 TARA_HCS1_H5_G100007472_G100006130--2013_10_31_18_54_20_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  59G Jul  9 04:09 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U00--V01.ome.zarr.zip
(base) [wmoore@pilot-zarr1-dev batch3]$ ls -alh
total 6.7G
drwxrwxr-x. 2 wmoore wmoore   4.0K Jul 10 07:30 idr0015
-rw-rw-r--. 1 wmoore wmoore   6.6G Jul  9 04:31 TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore    55M Jul  9 04:32 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01.ome.zarr.zip

Upload...

./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0015/batch3/idr0015 bsaspera_w@hx-fasp-1.ebi.ac.uk:5f/13xxxxxxx
will-moore commented 1 year ago

As suggested by Jason & Josh, looking at 7zip... https://www.7-zip.org/download.html

$ wget https://www.7-zip.org/a/7z2201-linux-x64.tar.xz
will-moore commented 1 year ago

Seems that we actually need p7zip, downloading from https://sourceforge.net/projects/p7zip/files/p7zip/16.02/

wget https://sourceforge.net/projects/p7zip/files/p7zip/16.02/p7zip_16.02_src_all.tar.bz2/download

That doesn't give me anything useful. Can't actually find any install instructions!

will-moore commented 1 year ago

Upload kinda slow (since they're not compressed):

(base) [wmoore@pilot-zarr1-dev bin]$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d /data/idr0015/batch3/idr0015 bsaspera_w@hx-fasp-1.ebi.ac.uk:5f/136e8d-xxxxxxxxx
TARA_HCS1_H5_G100004906_G100002201--2013_08_2                       100%   95GB  175Mb/s  1:00:51
TARA_HCS1_H5_G100006116_G100006253--2013_10_3                        100%   79GB 91.5Mb/s  1:52:26
TARA_HCS1_H5_G100006116_G100006253--2013_10_3                          100%   78GB  215Mb/s  2:52:26
TARA_HCS1_H5_G100006175_G100006179--2013_11_0                          100%   71GB  179Mb/s  3:38:16
TARA_HCS1_H5_G100006175_G100006179--2013_11_0                          100%   75GB  341Mb/s  4:27:16
TARA_HCS1_H5_G100007472_G100006130--2013_10_3                          100%   74GB  179Mb/s  5:07:43
TARA_HCS1_H5_G100007472_G100006130--2013_10_3                          100%   73GB  337Mb/s  5:46:02
TARA_HCS1_H5_G100007665_G100007576--2013_10_2                          100%   58GB  145Mb/s  6:15:54
will-moore commented 1 year ago

Batch4 (12 plates - including last 2 from batch3 which failed to complete)

mkdir batch4
for i in TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U01--V01 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U01--V01 TARA_HCS1_H5_G100008033_G100007130--2013_11_02_21_39_19_chamber--U00--V01 TARA_HCS1_H5_G100008033_G100007130--2013_11_02_21_39_19_chamber--U01--V01 TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01 TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U01--V01 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U00--V01 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U01--V01 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U00--V01 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U01--V01 TARA_HCS1_H5_G100008302_G100008304--2013_12_02_21_30_23_chamber--U00--V01; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen batch4/${i%.*}.ome.zarr; done
will-moore commented 1 year ago

Started uploading last completed zips from batch2:

(base) [wmoore@pilot-zarr1-dev bin]$ ls -lh /data/idr0015/batch2
total 423G
-rw-rw-r--. 1 wmoore wmoore  89G Jul 10 07:34 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  89G Jul 10 11:17 TARA_HCS1_H5_G100004339_G100004341--2013_09_27_15_24_28_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  72G Jul 10 15:03 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  72G Jul 10 18:22 TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore 103G Jul 10 22:53 TARA_HCS1_H5_G100004906_G100002201--2013_08_25_19_31_15_chamber--U00--V01.ome.zarr.zip
...
TARA_HCS1_H5_G100004339_G100004341--2013_09_2                        100%   88GB  165Mb/s    42:18
TARA_HCS1_H5_G100004339_G100004341--2013_09_2                         100%   88GB  352Mb/s  1:24:00
TARA_HCS1_H5_G100004727_G100004940--2013_12_0                          100%   71GB  242Mb/s  1:58:25
TARA_HCS1_H5_G100004727_G100004940--2013_12_0                          100%   71GB  271Mb/s  2:32:27
TARA_HCS1_H5_G100004906_G100002201--2013_08_2                         100%  102GB  231Mb/s  3:21:41

and batch1

(base) [wmoore@pilot-zarr1-dev bin]$ ls -lh /data/idr0015/batch1
total 485G
-rw-rw-r--. 1 wmoore wmoore  55G Jul 10 06:45 TARA_HCS1_H5_G100002655_G100002656--2013_09_24_15_21_06_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  55G Jul 10 09:10 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  55G Jul 10 12:14 TARA_HCS1_H5_G100002978_G100002980--2013_08_29_12_36_46_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore 143G Jul 10 18:06 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore 119G Jul 10 22:50 TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore  61G Jul 11 01:43 TARA_HCS1_H5_G100003584_G100003586--2013_09_26_15_23_10_chamber--U00--V01.ome.zarr.zip
...
TARA_HCS1_H5_G100002655_G100002656--2013_09_2                         100%   54GB  298Mb/s    25:40
TARA_HCS1_H5_G100002978_G100002980--2013_08_2                         100%   54GB  439Mb/s    50:48
TARA_HCS1_H5_G100002978_G100002980--2013_08_2                         100%   54GB  238Mb/s  1:15:34
TARA_HCS1_H5_G100003406_G100004906--2013_08_2                         100%  142GB  192Mb/s  2:20:05
TARA_HCS1_H5_G100003406_G100004906--2013_08_2                         100%  118GB  169Mb/s  3:15:32    
TARA_HCS1_H5_G100003584_G100003586--2013_09_2                          100%   60GB  196Mb/s  3:43:59    
Completed: 508294083K bytes transferred in 13440 seconds
 (309811K bits/sec), in 6 files, 1 directory.
will-moore commented 1 year ago

Zip batch4 (not all exported)

(base) [wmoore@pilot-zarr1-dev batch4]$ ls -lh 
total 40K
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 02:32 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 04:02 TARA_HCS1_H5_G100008033_G100007130--2013_11_02_21_39_19_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 05:34 TARA_HCS1_H5_G100008033_G100007130--2013_11_02_21_39_19_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 07:25 TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 08:43 TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 10:01 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 11:16 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 12:31 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 13:46 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 15:15 TARA_HCS1_H5_G100008302_G100008304--2013_12_02_21_30_23_chamber--U00--V01.ome.zarr

(base) [wmoore@pilot-zarr1-dev batch4]$ for i in */; do zip -mr "${i%/}.zip" "$i"; done
will-moore commented 1 year ago

10 hours later, only 2 zips created for batch4:

(base) [wmoore@pilot-zarr1-dev ~]$ ls -lh /data/idr0015/batch4
total 77G
-rw-rw-r--.  1 wmoore wmoore  35G Jul 11 20:41 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--.  1 wmoore wmoore  39G Jul 12 01:35 TARA_HCS1_H5_G100008033_G100007130--2013_11_02_21_39_19_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore 4.0K Jul 11 05:34 TARA_HCS1_H5_G100008033_G100007130--2013_11_02_21_39_19_chamber--U01--V01.ome.zarr
...

Not much space left:

(base) [wmoore@pilot-zarr1-dev ~]$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  4.8T  175G  97% /data
will-moore commented 1 year ago

Installed p7zip on pilot-zarr1-dev: https://github.com/IDR/idr-metadata/issues/651#issuecomment-1632133497

Current state of batch4 zipping...

(base) [wmoore@pilot-zarr1-dev ~]$ ls -alh /data/idr0015/batch4
total 136G
drwxrwxr-x.  9 wmoore wmoore   4.0K Jul 12 06:17 .
drwxrwxr-x.  5 wmoore dlindner  156 Jul 10 23:09 ..
-rw-rw-r--.  1 wmoore wmoore    35G Jul 11 20:41 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--.  1 wmoore wmoore    39G Jul 12 01:35 TARA_HCS1_H5_G100008033_G100007130--2013_11_02_21_39_19_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--.  1 wmoore wmoore    39G Jul 12 06:02 TARA_HCS1_H5_G100008033_G100007130--2013_11_02_21_39_19_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 07:25 TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 08:43 TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 10:01 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 11:16 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 12:31 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U00--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 13:46 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U01--V01.ome.zarr
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 15:15 TARA_HCS1_H5_G100008302_G100008304--2013_12_02_21_30_23_chamber--U00--V01.ome.zarr
-rw-------.  1 wmoore wmoore    25G Jul 12 09:08 ziIj7CU5

Cancelled that and try p7zip...

$ 7za a -sdel TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr.zip TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr
will-moore commented 1 year ago

moved that zip into batch4/idr0015 and upload... zip remaining batch 4 zarrs...

cd /data/idr0015/batch4
for i in */; do 7za a "${i%/}.zip" "$i"; done

Certainly a bit faster using 7za than regular zip... About 40 minutes per zip. But the originals aren't deleted.

total 267G
drwxrwxr-x.  9 wmoore wmoore   4.0K Jul 13 05:48 .
drwxrwxr-x.  7 wmoore dlindner  190 Jul 13 03:58 ..
drwxrwxr-x.  2 wmoore wmoore     10 Jul 13 03:48 idr0015
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 08:43 TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    46G Jul 13 03:13 TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 10:01 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U00--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    37G Jul 13 03:45 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 11:16 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    37G Jul 13 04:18 TARA_HCS1_H5_G100008152_G100008154--2013_11_04_20_31_35_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 12:31 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U00--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    47G Jul 13 05:00 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U00--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 13:46 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U01--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    47G Jul 13 05:40 TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U01--V01.ome.zarr.zip
drwxrwxr-x. 25 wmoore wmoore   4.0K Jul 11 15:15 TARA_HCS1_H5_G100008302_G100008304--2013_12_02_21_30_23_chamber--U00--V01.ome.zarr
-rw-rw-r--.  1 wmoore wmoore    56G Jul 13 06:28 TARA_HCS1_H5_G100008302_G100008304--2013_12_02_21_30_23_chamber--U00--V01.ome.zarr.zip

Uploading...

will-moore commented 1 year ago

batch5...

screen -r idr0015_ngff
mkdir batch5
mkdir memo
for i in TARA_HCS1_H5_G100008302_G100008304--2013_12_02_21_30_23_chamber--U01--V01 TARA_HCS1_H5_G100008608_G100008610--2013_11_05_20_28_35_chamber--U00--V01 TARA_HCS1_H5_G100008608_G100008610--2013_11_05_20_28_35_chamber--U01--V01 TARA_HCS1_H5_G100008990_G100009437--2013_12_07_22_33_03_chamber--U00--V01 TARA_HCS1_H5_G100008990_G100009437--2013_12_07_22_33_03_chamber--U01--V01 TARA_HCS1_H5_G100009617_G100008907--2013_12_06_20_46_40_chamber--U00--V01 TARA_HCS1_H5_G100009617_G100008907--2013_12_06_20_46_40_chamber--U01--V01 TARA_HCS1_H5_G100009725_G100010454--2013_12_05_21_08_22_chamber--U00--V01 TARA_HCS1_H5_G100009725_G100010454--2013_12_05_21_08_22_chamber--U01--V01 TARA_HCS1_H5_G100010173_G100010177--2013_08_22_17_12_07_chamber--U00--V01 TARA_HCS1_H5_G100010173_G100010177--2013_08_22_17_12_07_chamber--U01--V01; do echo $i; ~/bioformats2raw-0.6.0-24/bin/bioformats2raw --memo-directory ./memo /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen batch5/${i%.*}.ome.zarr; done
will-moore commented 1 year ago

Speed of zipping with 7zip...

45 - 50 minutes per zip:

(base) [wmoore@pilot-zarr1-dev idr0015]$ ls -lh
-rw-rw-r--.  1 wmoore wmoore  53G Jul 13 12:20 TARA_HCS1_H5_G100008302_G100008304--2013_12_02_21_30_23_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--.  1 wmoore wmoore  37G Jul 13 13:06 TARA_HCS1_H5_G100008608_G100008610--2013_11_05_20_28_35_chamber--U00--V01.ome.zarr.zip
-rw-rw-r--.  1 wmoore wmoore  38G Jul 13 13:47 TARA_HCS1_H5_G100008608_G100008610--2013_11_05_20_28_35_chamber--U01--V01.ome.zarr.zip
-rw-rw-r--.  1 wmoore wmoore  41G Jul 13 14:32 TARA_HCS1_H5_G100008990_G100009437--2013_12_07_22_33_03_chamber--U00--V01.ome.zarr.zip
will-moore commented 1 year ago

Convert last 25 (still need to go back and fix 3 that failed earlier)... Not a ton of space...

ssh pilot-zarr1-dev

$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        4.9T  4.2T  713G  86% /data

cd /data
mkdir idr0015 && cd idr0015
mkdir batch6
mkdir memo
screen -r idr0015_ngff

for i in TARA_HCS1_H5_G100010173_G100010177--2014_06_24_12_25_12_chamber--U00--V01 TARA_HCS1_H5_G100010173_G100010177--2014_06_24_12_25_12_chamber--U01--V01 TARA_HCS1_H5_G100010237_G100010241--2013_08_20_15_53_43_chamber--U00--V01 TARA_HCS1_H5_G100010237_G100010241--2013_08_20_15_53_43_chamber--U01--V01 TARA_HCS1_H5_G100010237_G100010241--2014_06_25_13_42_23_chamber--U00--V01 TARA_HCS1_H5_G100010237_G100010241--2014_06_25_13_42_23_chamber--U01--V01 TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01 TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U01--V01 TARA_HCS1_H5_G100010607_G100010623--2014_06_23_16_05_11_chamber--U00--V01 TARA_HCS1_H5_G100010607_G100010623--2014_06_23_16_05_11_chamber--U01--V01 TARA_HCS1_H5_G100010623_G100010731--2013_08_23_17_27_23_chamber--U00--V01 TARA_HCS1_H5_G100010623_G100010731--2013_08_23_17_27_23_chamber--U01--V01 TARA_HCS1_H5_G100010623_G100010731--2014_06_20_18_48_17_chamber--U00--V01 TARA_HCS1_H5_G100010623_G100010731--2014_06_20_18_48_17_chamber--U01--V01 TARA_HCS1_H5_G100010824_G100010826--2013_12_03_22_14_42_chamber--U00--V01 TARA_HCS1_H5_G100010824_G100010826--2013_12_03_22_14_42_chamber--U01--V01 TARA_HCS1_H5_G100010891_G100010893--2013_12_04_21_23_54_chamber--U00--V01 TARA_HCS1_H5_G100010891_G100010893--2013_12_04_21_23_54_chamber--U01--V01 TARA_HCS1_H5_G100011084_G100010958--2013_10_04_09_31_04_chamber--U00--V01 TARA_HCS1_H5_G100011084_G100010958--2013_10_04_09_31_04_chamber--U01--V01 TARA_HCS1_H5_G100011162_G100012868--2013_10_03_13_08_50_chamber--U00--V01 TARA_HCS1_H5_G100011162_G100012868--2013_10_03_13_08_50_chamber--U01--V01 TARA_HCS1_H5_G100012537_G100012477--2013_10_01_12_30_33_chamber--U00--V01 TARA_HCS1_H5_G100012537_G100012477--2013_10_01_12_30_33_chamber--U01--V01 TARA_HCS1_H5_G100012694_G100012776--2013_10_02_14_27_33_chamber--U00--V01 TARA_HCS1_H5_G100012694_G100012776--2013_10_02_14_27_33_chamber--U01--V01; do
  ~/bioformats2raw-0.6.0-24/bin/bioformats2raw --memo-directory ./memo /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen batch6/${i%.*}.ome.zarr; done
will-moore commented 1 year ago

Need a simple script to process zarrs generated above:

Creates it's own idr0015 dir which we know is empty

create script zip_upload.sh in /data/idr0015/upload/zip_upload.sh

mkdir idr0015
for i in `ls | grep zarr`; do
  echo $i;
  7za a "idr0015/${i%/}.zip" "$i";
  ~/.aspera/cli/bin/ascp -P33001 -i ~/.aspera/cli/etc/asperaweb_id_dsa.openssh -d idr0015 bsaspera_w@hx-fasp-1.ebi.ac.uk:5f/136xxxxx
  rm -rf "idr0015/${i%/}.zip"
  # rm -rf "$i"
done

Then we can move files into /upload/ etc...

mv *.zarr upload
cd upload
./zip_upload.sh

Seems to work OK, but this plate didn't fully export (so the zip is small). Need to re-export.

TARA_HCS1_H5_G100010241_G100010731--2013_09_29_19_14_59_chamber--U00--V01.ome.zarr
will-moore commented 1 year ago

However.... using the zip_upload.sh script above takes a long time before anything is deleted, and runs all the steps for each zarr in series. Useful to process a bunch of data for longer periods without tending to the process, but not as fast as manually running zip, upload and delete steps in parallel as above, so will continue with this approach...

will-moore commented 1 year ago

https://www.ebi.ac.uk/biostudies/submissions/files?path=%2Fuser%2Fidr0015 lists 82 plates, but we need 84. To find missing plates, use JS Console to load names from IDR and compare...

let url = "https://idr.openmicroscopy.org/webclient/api/plates/?id=1201"
let idr_plates = await fetch(url).then(rsp => rsp.json());
let idr_names = idr_plates.plates.map(p => p.name);
let names = [];
[].forEach.call(document.querySelectorAll("div [role='row'] .ag-cell[col-id='name']"), function(div) {
  names.push(div.innerHTML.trim().replace(".ome.zarr.zip", ""));
});
idr_names.forEach(n => {if (names.indexOf(n) == -1) {console.log(n)}; });
  TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U01--V01
  TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01
will-moore commented 1 year ago

2 plates missing from batch3 above. Re-export...

screen -r idr0015_ngff
cd /data/idr0015/batch3
for i in TARA_HCS1_H5_G100007665_G100007576--2013_10_28_21_05_26_chamber--U01--V01 TARA_HCS1_H5_G100007887_G100007889--2013_10_29_19_56_21_chamber--U00--V01; do
  ~/bioformats2raw-0.6.0-24/bin/bioformats2raw --memo-directory ./memo /uod/idr/metadata/idr0015-colin-taraoceans/screenA/patterns/$i.screen batch6/${i%.*}.ome.zarr; done
will-moore commented 1 year ago

Moved those 2 .zarrs into upload dir, then...

cd upload
./zip_upload.sh
will-moore commented 1 year ago

22 out of 84 filesets "viewable" at https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD861.html

On idr0125-pilot, take first 10 for testing...

idr0015/TARA_HCS1_H5_G100010607_G100010623--2014_06_23_16_05_11_chamber--U01--V01.ome.zarr,S-BIAD861/00fc2a08-e352-4720-beac-13fd06cda6b2,21164
idr0015/TARA_HCS1_H5_G100010237_G100010241--2014_06_25_13_42_23_chamber--U00--V01.ome.zarr,S-BIAD861/0fb7c57c-fc23-43a0-b5aa-b12d691784b4,21159
idr0015/TARA_HCS1_H5_G100006175_G100006179--2013_11_01_23_35_08_chamber--U01--V01.ome.zarr,S-BIAD861/17a8eb33-2684-40af-9801-c25254537e92,21109
idr0015/TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01.ome.zarr,S-BIAD861/17b51dec-bd98-44af-8041-0e4615e755d0,21051
idr0015/TARA_HCS1_H5_G100006116_G100006253--2013_10_30_19_38_12_chamber--U01--V01.ome.zarr,S-BIAD861/31a566c3-55a9-4efc-8565-b711a1b0bf6b,21107
idr0015/TARA_HCS1_H5_G100008236_G100008238--2013_11_06_21_52_31_chamber--U01--V01.ome.zarr,S-BIAD861/39063bfc-ff55-4f8e-a062-1ae1e8da24a2,21123
idr0015/TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U00--V01.ome.zarr,S-BIAD861/48eb9947-fb37-417a-bbb7-cda59f112229,20306
idr0015/TARA_HCS1_H5_G100010623_G100010731--2014_06_20_18_48_17_chamber--U01--V01.ome.zarr,S-BIAD861/4b1f9efb-965a-481f-bafc-195dc96c1219,21168
idr0015/TARA_HCS1_H5_G100008033_G100007130--2013_11_02_21_39_19_chamber--U00--V01.ome.zarr,S-BIAD861/67e274a4-78db-4ace-bb66-d2010ec9891b,21116
idr0015/TARA_HCS1_H5_G100003406_G100004906--2013_08_24_19_23_14_chamber--U01--V01.ome.zarr,S-BIAD861/73d2e66a-a737-46b8-b174-6d60e9145b45,20903
Found prefix demo_2/2016-06/11 // 02-07-11.805 for fileset 21164
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/11/02-07-11.805
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/11/02-07-11.805_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/11/02-07-11.805_mkngff/00fc2a08-e352-4720-beac-13fd06cda6b2.zarr -> /bia-integrator-data/S-BIAD861/00fc2a08-e352-4720-beac-13fd06cda6b2/00fc2a08-e352-4720-beac-13fd06cda6b2.zarr
...
will-moore commented 1 year ago

Re-ran on idr0138-pilot with the same inputs, but server must have restarted at some point because the last few didn't get logged-in. Then I needed to manually update the sql scripts with a new SECRET, so I only did the first 3 above...

BEGIN
 mkngff_fileset 
----------------
        5811649
(1 row)

COMMIT

e.g. 3rd plate above... Waiting on memo file...

http://localhost:1040/webclient/?show=image-1961268

will-moore commented 1 year ago

In IDR, we have 2 duplicate Plates named TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01 https://idr.openmicroscopy.org/webclient/?show=plate-4801 https://idr.openmicroscopy.org/webclient/?show=plate-4751

but we have NO plate named TARA_HCS1_H5_G100008990_G100009437--2013_12_07_22_33_03_chamber--U00--V01 which exists on disk (and we have generated NGFF plate for it: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD861/9b150385-9a50-4454-9a4e-18c477f938ea/9b150385-9a50-4454-9a4e-18c477f938ea.zarr/

Screenshot 2023-09-22 at 15 31 22

We can't use the same NGFF plate for the new Filesets of both duplicate plates, since OMERO doesn't allow duplicate path/names in OriginalFiles table. Probably easiest to simply delete one of the Plates (as we did with idr0004).

will-moore commented 1 year ago

https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD861.html still has 1 Plate not on s3: IM30 | idr0015/TARA_HCS1_H5_G100008060_G100008062--2013_11_03_22_51_32_chamber--U00--V01.ome.zarr.zip

Will need to update later...

Running on idr0125-pilot... Using idr0015.csv without the incomplete last line...

(venv3) (base) bash-4.2$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql $fsid "/bia-integrator-data/$biapath/$uuid.zarr" >> "$IDRID/$fsid.sql"
> done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/2016-06/09/07-02-56.345 for fileset: 21152
will-moore commented 1 year ago

Failed due to goofys after 16 Filesets. Remounted, edited idr0015.csv to remove 1st 16 lines and restarted... (on idr0125-pilot)...

(venv3) (base) bash-4.2$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql $fsid "/bia-integrator-data/$biapath/$uuid.zarr" >> "$IDRID/$fsid.sql"
> done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/2016-06/05/09-15-45.736 for fileset: 21107