search

hputnam / Geoduck_Meth

DNA methylation of geoduck exposed to ocean acidification treatments
1 stars 0 forks source link

Update OSF for trimmed reads and everything post alignment #4

Closed hputnam closed 4 years ago

hputnam commented 4 years ago

https://gannet.fish.washington.edu/seashell/bu-mox/scrubbed/032120-fds/

https://osf.io/yem8n/

sr320 commented 4 years ago

transferring files to desktop

sr320 commented 4 years ago

new sorted bams and indices are now updated

sr320 commented 4 years ago

5x and 10 bedgraphs updated

shellywanamaker commented 4 years ago

@sr320 @kubu4 ever got tried this for uploading files? https://osfclient.readthedocs.io/en/latest/

sr320 commented 4 years ago

I have not - are you using it? On Mar 29, 2020, 9:54 AM -0700, Shelly Trigg notifications@github.com, wrote:

@sr320 @kubu4 ever got tried this for uploading files? https://osfclient.readthedocs.io/en/latest/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

kubu4 commented 4 years ago

Also have not used it.

shellywanamaker commented 4 years ago

I tried but I keep getting this error:

osf -p yem8n -u strigg@uw.edu upload /Volumes/web/metacarcinus/Pgenerosa/analyses/20200320/TG_FASTQS/EPI-41_S38_L005_R1_001_val_1.fq.gz osfstorage/trimmed
Please input your password: 
usage: osf upload [-h] [-f] [-r] source destination
osf upload: error: You are not authorized to access this project.

I even tried changing the the project write access settings to all OSF users and that still didn't resolve the problem (it's now switched back to contributors). After looking through the issues for that software it doesn't seem very robust and doesn't appear to have good support https://github.com/osfclient/osfclient/issues.

It's going to take me a while to upload the trimmed reads (96GB). I'm currently copying files to my desktop from Gannet (~4 min/file) and have to do it in batches because i don't have enough local space to do it all at once. And uploading one file at a time.

How do we check the integrity of the files uploaded?

shellywanamaker commented 4 years ago

also @kubu4 @sr320 what do you think is the most efficient way to upload files to OSF?

shellywanamaker commented 4 years ago

it's ~20min to upload 1 file

sr320 commented 4 years ago

I can get the trimmed files up no problem…

Also it is probably worth setting up a way to work on campus machines remotely.

On Mar 29, 2020, 10:16 AM -0700, Shelly Trigg notifications@github.com, wrote:

I tried but I keep getting this error: osf -p yem8n -u strigg@uw.edu upload /Volumes/web/metacarcinus/Pgenerosa/analyses/20200320/TG_FASTQS/EPI-41_S38_L005_R1_001_val_1.fq.gz osfstorage/trimmed Please input your password: usage: osf upload [-h] [-f] [-r] source destination osf upload: error: You are not authorized to access this project. I even tried changing the the project write access settings to all OSF users and that still didn't resolve the problem (it's now switched back to contributors). After looking through the issues for that software it doesn't seem very robust and doesn't appear to have good support https://github.com/osfclient/osfclient/issues. It's going to take me a while to upload the trimmed reads (96GB). I'm currently copying files to my desktop from Gannet (~4 min/file) and have to do it in batches because i don't have enough local space to do it all at once. And uploading one file at a time. How do we check the integrity of the files uploaded? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

sr320 commented 4 years ago

Use that time to wrap up…..

the oyster intro.

On Mar 29, 2020, 10:19 AM -0700, Steven Roberts roberts.sbr@gmail.com, wrote:

I can get the trimmed files up no problem…

Also it is probably worth setting up a way to work on campus machines remotely.

On Mar 29, 2020, 10:16 AM -0700, Shelly Trigg notifications@github.com, wrote:

I tried but I keep getting this error: osf -p yem8n -u strigg@uw.edu upload /Volumes/web/metacarcinus/Pgenerosa/analyses/20200320/TG_FASTQS/EPI-41_S38_L005_R1_001_val_1.fq.gz osfstorage/trimmed Please input your password: usage: osf upload [-h] [-f] [-r] source destination osf upload: error: You are not authorized to access this project. I even tried changing the the project write access settings to all OSF users and that still didn't resolve the problem (it's now switched back to contributors). After looking through the issues for that software it doesn't seem very robust and doesn't appear to have good support https://github.com/osfclient/osfclient/issues. It's going to take me a while to upload the trimmed reads (96GB). I'm currently copying files to my desktop from Gannet (~4 min/file) and have to do it in batches because i don't have enough local space to do it all at once. And uploading one file at a time. How do we check the integrity of the files uploaded? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

sr320 commented 4 years ago

Also do feel free to upgrade your local space as Sam did.

On Mar 29, 2020, 10:19 AM -0700, Steven Roberts roberts.sbr@gmail.com, wrote:

Use that time to wrap up…..

the oyster intro.

On Mar 29, 2020, 10:19 AM -0700, Steven Roberts roberts.sbr@gmail.com, wrote:

I can get the trimmed files up no problem…

Also it is probably worth setting up a way to work on campus machines remotely.

On Mar 29, 2020, 10:16 AM -0700, Shelly Trigg notifications@github.com, wrote:

I tried but I keep getting this error: osf -p yem8n -u strigg@uw.edu upload /Volumes/web/metacarcinus/Pgenerosa/analyses/20200320/TG_FASTQS/EPI-41_S38_L005_R1_001_val_1.fq.gz osfstorage/trimmed Please input your password: usage: osf upload [-h] [-f] [-r] source destination osf upload: error: You are not authorized to access this project. I even tried changing the the project write access settings to all OSF users and that still didn't resolve the problem (it's now switched back to contributors). After looking through the issues for that software it doesn't seem very robust and doesn't appear to have good support https://github.com/osfclient/osfclient/issues. It's going to take me a while to upload the trimmed reads (96GB). I'm currently copying files to my desktop from Gannet (~4 min/file) and have to do it in batches because i don't have enough local space to do it all at once. And uploading one file at a time. How do we check the integrity of the files uploaded? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

shellywanamaker commented 4 years ago

@sr320 that would be awesome if you could upload the trimmed data!

it's all the *.fq.gz here: https://gannet.fish.washington.edu/metacarcinus/Pgenerosa/analyses/20200320/TG_FASTQS/

sr320 commented 4 years ago

Actually I used /gscratch/scrubbed/strigg/analyses/20200320/TG_FASTQS/

On Mar 29, 2020, 10:27 AM -0700, Shelly Trigg notifications@github.com, wrote:

@sr320 that would be awesome if you could upload the trimmed data! it's all the *.fq.gz here: https://gannet.fish.washington.edu/metacarcinus/Pgenerosa/analyses/20200320/TG_FASTQS/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

shellywanamaker commented 4 years ago

I can remote onto the campus machines via terminal, but since can't transfer files of OSF via command line that doesn't help. When I'm on campus I can use the screen sharing application to connect to ostrich for GUI access but I can't get it to connect from home.

What are you thinking in terms of "setting up a way to work on campus machines remotely."?

/gscratch/scrubbed/strigg/analyses/20200320/TG_FASTQS/ is fine (it's the same contents as https://gannet.fish.washington.edu/metacarcinus/Pgenerosa/analyses/20200320/TG_FASTQS/

sr320 commented 4 years ago

I find GUI remote very handy on a daily basis using Remote Desktop I use it of independent rsync operations, OSF uploads, downloading UW restricted content…

On Mar 29, 2020, 10:31 AM -0700, Shelly Trigg notifications@github.com, wrote:

I can remote onto the campus machines via terminal, but since can't transfer files of OSF via command line that doesn't help. When I'm on campus I can use the screen sharing application to connect to ostrich for GUI access but I can't get it to connect from home. What are you thinking in terms of "setting up a way to work on campus machines remotely."? /gscratch/scrubbed/strigg/analyses/20200320/TG_FASTQS/ is fine (it's the same contents as https://gannet.fish.washington.edu/metacarcinus/Pgenerosa/analyses/20200320/TG_FASTQS/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

sr320 commented 4 years ago

Update: bismark coverage files uploaded

shellywanamaker commented 4 years ago

what's Remote Desktop?

sr320 commented 4 years ago

see https://github.com/RobertsLab/resources/issues/876

On Mar 29, 2020, 10:41 AM -0700, Shelly Trigg notifications@github.com, wrote:

what's Remote Desktop? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

sr320 commented 4 years ago

trimmed reads, merged 5x tab, coverage files, bams, and bedgraphs updated.