External upload links for large volumes (800 GB)

[ofrei]

Our collaborators struggle to upload large volumes via external upload links - we need to discuss if there are alternative solutions to this.

[ofrei]

TSD documentation about external import grants: https://www.uio.no/english/services/it/research/sensitive-data/use-tsd/import-export/?chatbot#toc5

[ofrei]

External import is not for large scale data at the moment. It should work well up to 10 GB of data, but 100 GB would be too large. This data we need to transfer via rsync to NIRD (encrypted), and then import to TSD ourself.

[ofrei]

@Sandeek Could you please handle the latest transfer from deCODE?

[ofrei]

Technical update (the way I see this):

• it seem not straightforward to do the transfer at USIT TSD, because only physical Linux machines are available; we need windows because the disk is bitlocker-encrypted
• we can't handle it via web upload ( http://data.tsd.usit.no/ ) because of the size of the data (1.5 TB in total)
• it would be difficult to move the data to linux envirnment because all files on disk are very large (up to 200 GB), so we can't use another drive with FAT32 file system - while NFTS filesystem isn't easy to mount on Linux
• the most viable option is to use "rsync" from WSL. For a stable rsync it's important to be on a physical network. Pain point here is that (1) we can't have Admin rights on Windows machines if they are deployed by USIT, and (2) only machines deployed by USIT are recognized by UiO network.

@Sandeek Just ping me if you need help!

[Sandeek]

@ofrei should we push Tor for the physical setup of the machine or we continue with data transfer with the help of USIT?

[ofrei]

@Sandeek It's best to have a local desktop at your workplace with Fedora (UiO-supported image in linux). We can still reach out for help from TSD for 10 TB+ large transfers, but below that we should be able to do ourselves, I think.