search

davidpng / FCS_Database

Program to scrape an FCS directory of metadata
GNU General Public License v3.0
3 stars 2 forks source link

Raw .fcs data #40

Open hermands opened 9 years ago

hermands commented 9 years ago

Plan: (1) Wait until Jerry has made full 64 TB virtual (2) Wait until Ed/Tyler have built directory structure and permission scripts (3) Hook up USB/laptop to switch, transfer data to /net (4) Attach drive to larf/yeti
! make sure stable! Format drive (6) Copy data from /net to drive

DO STUFF!

davidpng commented 9 years ago

At Tuesday lab meeting, it was decided to start moving things to the SFS.

Currently moving data to /net/emc/projects/dan_herman_project/FCS_Files/ @ 35 MB/sec

davidpng commented 9 years ago

I found out that SCP is the bottleneck. We can transfer multiple folders with multiple instances to SCP to fill the pipes. Another option is using bbcp brew install bbcp

However this will also need to be installed in the remote machine (Yeti). Perhaps talk with Noah or Ed?

hermands commented 9 years ago

Sounds pretty slow. Hmm...If we want it installed on yeti, I would talk with Ed/Sheena/Tyler. Is there another built-in protocol that would be faster? rsync is the same issue? You are connected directly to the switch?

davidpng commented 9 years ago

rsync has similar issues, I've open up multiple scp instances and I"m getting 80 MB/sec which is the close the max drive speed or 1GBPS speed. So we will see.

hermands commented 9 years ago

Sounds good

hermands commented 9 years ago

Data is now on /net/emc. Plan is to wait until shared file system is stable. Then delete USB drive data, reformat, attach to larf/yeti. Copy data over to new USB drive.

hermands commented 9 years ago

Need a direct rsync from SCCA and yeti to clean up.