search

pelkmanslab / IT_Infra_Issues_of_the_lab

Use issue tracker to report general lab problem
0 stars 0 forks source link

Migrations of data from UZH to FMI (Liberali lab) #6

Open ewiger opened 8 years ago

ewiger commented 8 years ago

There are kinds of data (produced by Prisca in the lab) to be migrated:

_Endocytome_ dataset is located at ./camelot-share3/Data/Datasets/endocytome. This can be copied already now.

Unfortunately the situation with archive data is more complicated. There are some redundant and old copies that eventually can be removed. All folders with name Prisca are distributed over the following locations:

./pelkmanslab-share1/Data/Users/Prisca
./camelot-share2/Data/Users/Prisca
./camelot-share3/Data/Users/Prisca
./ethz-share4/Data/Users/Prisca
./ethz-share5/Data/Users/Prisca
./ethz-share5/Archive/Berend/Prisca
./camelot-share1/Data/Users/Prisca

However this list might be incomplete. But clearly all mentioned folders have to be copied to FMI. At the destination sites copied data will keep the names of NAS shares. This will help to merge folders and identify the duplicates.

We will use SSH access to camelot server as a gateway for all copying operations.

ewiger commented 8 years ago 
6.1T    endocytome
SaadiaIftikhar commented 8 years ago

I have restarted Prisca data, there are some files missing which needs special handling as files has spaces in the names ...

These files are:

t_A431 reg.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/WASL 14-3, CH 6 log MacroVesMeanInt_A431.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/WDR44 14-4, CH 6 log MacroVesMeanInt_A431 reg.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/WDR44 14-4, CH 6 log MacroVesMeanInt_A431.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/WIPF1 14-5, CH 6 log MacroVesMeanInt_A431 reg.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/WIPF1 14-5, CH 6 log MacroVesMeanInt_A431.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/WIPF2 14-6, CH 6 log MacroVesMeanInt_A431 reg.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/WIPF2 14-6, CH 6 log MacroVesMeanInt_A431.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/ZFYVE16 14-7, CH 6 log MacroVesMeanInt_A431 reg.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/ZFYVE16 14-7, CH 6 log MacroVesMeanInt_A431.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/ZFYVE9 14-8, CH 6 log MacroVesMeanInt_A431 reg.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/ZFYVE9 14-8, CH 6 log MacroVesMeanInt_A431.png": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/controls.mat": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/cyt MacroVesMeanInt_A431_100402_A431_Macropinocytosis_CP395-1bd.mat": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/removed_weights.mat": Permission denied (13) rsync: send_files failed to open "/share/nas/camelot-share2/Data/Users/Prisca/150316_FullTrajectoryAnalysis/PartialPenetrance/endocytome/MacroVesMeanInt_A431/CP395-1bd/traj vec.mat": Permission denied (13) rsync: opendir "/share/nas/camelot-share2/Data/Users/Prisca/LabMeeting/ProgressReport" failed: Permission denied (13) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [generator=3.1.0] copying camelot.pelkmanslab.org:/share/nas/camelot-share3/Data/Users/Prisca/ -> /share/nas/fmi_liberali-share1/MOVE_UZH/all_of_Prisca/camelot-share3/Prisca

some logs:

receiving incremental file list

Number of files: 828,505 (reg: 828,341, dir: 164) Number of created files: 0 Number of regular files transferred: 0 Total file size: 433,770,748,270 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 57,789,012 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 174 Total bytes received: 57,920,504

sent 174 bytes received 57,920,504 bytes 100,819.28 bytes/sec total size is 433,770,748,270 speedup is 7,489.05

receiving incremental file list

Number of files: 1,293,882 (reg: 1,292,255, dir: 1,627) Number of created files: 0 Number of regular files transferred: 0 Total file size: 1,846,895,722,737 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 53,801,851 File list generation time: 0.005 seconds File list transfer time: 0.000 seconds Total bytes sent: 1,500,171 Total bytes received: 73,052,003

sent 1,500,171 bytes received 73,052,003 bytes 94,669.43 bytes/sec total size is 1,846,895,722,737 speedup is 24,773.20

and camelot share 3 has a 0kb log file in it.

These all logs are updated today after lunch.

For endocytomes:

log files isn't opening to check the status ...

SaadiaIftikhar commented 8 years ago

endocytose camelot-share3 and camelot-share3_Prisca still in the progress. so have to wait for sometime.

camelot-share1_Prisca and camelot-share2_Prisca are done. I think.

Best Saadia

SaadiaIftikhar commented 8 years ago

share1-Prisca

receiving incremental file list

Number of files: 828,505 (reg: 828,341, dir: 164) Number of created files: 0 Number of regular files transferred: 0 Total file size: 433,770,748,270 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 57,789,012 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 174 Total bytes received: 57,920,504

sent 174 bytes received 57,920,504 bytes 100,819.28 bytes/sec total size is 433,770,748,270 speedup is 7,489.05

share2-Prisca

receiving incremental file list

Number of files: 1,293,882 (reg: 1,292,255, dir: 1,627) Number of created files: 0 Number of regular files transferred: 0 Total file size: 1,846,895,722,737 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 53,801,851 File list generation time: 0.005 seconds File list transfer time: 0.000 seconds Total bytes sent: 1,500,171 Total bytes received: 73,052,003

sent 1,500,171 bytes received 73,052,003 bytes 94,669.43 bytes/sec total size is 1,846,895,722,737 speedup is 24,773.20

share3-Prisca

receiving incremental file list

Number of files: 8,083,343 (reg: 8,043,412, dir: 39,931) Number of created files: 0 Number of regular files transferred: 0 Total file size: 8,286,423,633,795 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 332,071,506 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 39,941 Total bytes received: 334,767,926

sent 39,941 bytes received 334,767,926 bytes 25,061.41 bytes/sec total size is 8,286,423,633,795 speedup is 24,749.79

SaadiaIftikhar commented 8 years ago

endocytose:

Number of files: 7,534,197 (reg: 7,532,860, dir: 1,337) Number of created files: 0 Number of regular files transferred: 1,065 Total file size: 6,653,807,803,331 bytes Total transferred file size: 1,311,956,455 bytes Literal data: 0 bytes Matched data: 1,311,956,455 bytes File list size: 316,898,941 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 6,486,620 Total bytes received: 322,018,074

sent 6,486,620 bytes received 322,018,074 bytes 26,245.73 bytes/sec total size is 6,653,807,803,331 speedup is 20,254.83

SaadiaIftikhar commented 8 years ago

endocytome:

receiving incremental file list endocytome_FollowUps/ endocytome_FollowUps_hardlinks01/ endocytome_LN02/

Number of files: 7,534,197 (reg: 7,532,860, dir: 1,337) Number of created files: 0 Number of regular files transferred: 0 Total file size: 6,653,807,803,331 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 316,122,306 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 1,368 Total bytes received: 317,665,870

sent 1,368 bytes received 317,665,870 bytes 16,387.69 bytes/sec total size is 6,653,807,803,331 speedup is 20,945.84

ewiger commented 8 years ago

Very nice. All the data is copied. Now we can finish on deciding what to archive, what to keep and what to trash. @SaadiaIftikhar big thanks!