Raven drive full?

[sr320]

I think /home/shared/8TB_HDD_01 is stuffed as not even showing up with command below

df -h | grep -E "/dev/sd[abdef]1"
/dev/sde1       916G   15M  870G   1% /media/1TB_ext_SSD01
/dev/sdf1       916G   12K  870G   1% /media/1TB_ext_SSD02
/dev/sdd1       7.3T  3.3T  3.7T  48% /home/shared/8TB_HDD_02
/dev/sda1       771M   55M  717M   8% /boot/efi
/dev/sdb1       916G   60K  870G   1% /home/shared/1TB_HDD02

[kubu4]

Are you experiencing an error?

Here's what I see for the two 8TB HDDs:

[sr320]

Repo / Rstudio was non responsive - I went in and deleted a bunch of files.

Would love to get another drive accessible

On Thu, Jul 13, 2023 at 8:46 AM kubu4 @.***> wrote:

Are you experiencing an error?

Here's what I see for the two 8TB HDDs:

[image: image] https://urldefense.com/v3/__https://user-images.githubusercontent.com/4514104/253333071-ef8a35bc-d7f0-4c35-9c49-0cbd81011832.png__;!!K-Hz7m0Vt54!gSqfExFaXoVeySRJdDGBxJJJFo_HEt6v2FjqvFjPxLE2ED3Gt5BwRT7sWt6VF854rHV1D-iuLvEKMCYVfdJGbO0$

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/RobertsLab/resources/issues/1677*issuecomment-1634483681__;Iw!!K-Hz7m0Vt54!gSqfExFaXoVeySRJdDGBxJJJFo_HEt6v2FjqvFjPxLE2ED3Gt5BwRT7sWt6VF854rHV1D-iuLvEKMCYVEXK7ry4$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABB4PN7MZ4VVJGEE2DNY52LXQAJ5LANCNFSM6AAAAAA2JCNHFY__;!!K-Hz7m0Vt54!gSqfExFaXoVeySRJdDGBxJJJFo_HEt6v2FjqvFjPxLE2ED3Gt5BwRT7sWt6VF854rHV1D-iuLvEKMCYV-vLKOjE$ . You are receiving this because you authored the thread.Message ID: @.***>

--

Steven B. Roberts, Professor School of Aquatic and Fishery Sciences University of Washington Fisheries Teaching and Research (FTR) Building - Office 232 1140 NE Boat Street - Seattle, WA 98105 robertslab.info - @.*** - @sr320 vm:206.866.5141 - cell:360.362.3626 schedule a zoom call: https://d.pr/gsgxVJ

[kubu4]

Would love to get another drive accessible

There are those other three 1TB drives which no one is using.

Could probably swap out the internal 1TB for an 8TB HDD...

[kubu4]

We've run out of space on Raven. Here are the big users:

1.3T    ./mattgeorgephd
1.2T    ./zbengt
889G    ./graceleuchtenberger
714G    ./graceac9

If some of you could reevaluate your space needs and delete stuff, that would be helpful. Thanks @mattgeorgephd , @grace-ac, @zbengt , @graceleuchtenberger.

You can always back up to Gannet, delete from Raven, and then move things back as needed.

[graceleuchtenberger]

Just realized I had two copies of my data in my Github. Deleted the duplicates. Hopefully that helps.

[grace-ac]

had some extra directories with trimmed and untrimmed rnaseq data - deleted extras!

[zbengt]

Just deleted some big sam and fastq files!

[kubu4]

Thanks. That should help for a little bit. @zbengt - You're still consuming 1.1TB of space. If you can find more stuff to groom, that'd be killer. However, I'll close this for now. Thanks again for your help!

[kubu4]

Re-opening this, per this comment.

Here are the top space users:

@mattgeorgephd - I feel like this comes up with some regularity and your space usage is always the most, and never changes. Any chance you can delete some data?

@grace-ac - I know you actively have things going on, but is there any stuff you could possible get rid of?

@meganewing - Another option for you would be to modify your code so that you're deleting each SAM file immediately after you convert each to BAM. That way, the SAM files don't accumulate (they eat up LOTS of disk space).

[sr320]

We have space on drive 03? Let’s just port Matt’s stuff there - he is so close to getting that methylation paper done ;)

On Sat, Apr 20, 2024 at 3:41 PM kubu4 @.***> wrote:

Re-opening this, per this comment https://urldefense.com/v3/__https://github.com/RobertsLab/resources/issues/1876*issuecomment-2067803299__;Iw!!K-Hz7m0Vt54!iZqe7-AnI4Vtugw9afFeAQd6uwH4w_TXZZWzkolgttN3SzINRmFj2Llova1flaJE6QxRLHYqleLqQy9RwQ1bR90$ .

Here are the top space users:

image.png (view on web) https://urldefense.com/v3/__https://github.com/RobertsLab/resources/assets/4514104/8ec8d3c9-7fde-4d64-9105-c8834dbb04e9__;!!K-Hz7m0Vt54!iZqe7-AnI4Vtugw9afFeAQd6uwH4w_TXZZWzkolgttN3SzINRmFj2Llova1flaJE6QxRLHYqleLqQy9RsvNWPl4$

@mattgeorgephd https://urldefense.com/v3/__https://github.com/mattgeorgephd__;!!K-Hz7m0Vt54!iZqe7-AnI4Vtugw9afFeAQd6uwH4w_TXZZWzkolgttN3SzINRmFj2Llova1flaJE6QxRLHYqleLqQy9RDO1-57g$

• I feel like this comes up with some regularity and your space usage is always the most, and never changes. Any chance you can delete some data?

@grace-ac https://urldefense.com/v3/__https://github.com/grace-ac__;!!K-Hz7m0Vt54!iZqe7-AnI4Vtugw9afFeAQd6uwH4w_TXZZWzkolgttN3SzINRmFj2Llova1flaJE6QxRLHYqleLqQy9Ry583FX4$

• I know you actively have things going on, but is there any stuff you could possible get rid of?

@meganewing https://urldefense.com/v3/__https://github.com/meganewing__;!!K-Hz7m0Vt54!iZqe7-AnI4Vtugw9afFeAQd6uwH4w_TXZZWzkolgttN3SzINRmFj2Llova1flaJE6QxRLHYqleLqQy9RmLzH6_c$

• Another option for you would be to modify your code so that you're deleting each SAM file immediately after you convert each to BAM. That way, the SAM files don't accumulate (they eat up LOTS of disk space).

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/RobertsLab/resources/issues/1677*issuecomment-2067805818__;Iw!!K-Hz7m0Vt54!iZqe7-AnI4Vtugw9afFeAQd6uwH4w_TXZZWzkolgttN3SzINRmFj2Llova1flaJE6QxRLHYqleLqQy9RFzQCnG0$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABB4PNYF6QKLTRN5ROPH5GLY6LVKZAVCNFSM6AAAAAA2JCNHF2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRXHAYDKOBRHA__;!!K-Hz7m0Vt54!iZqe7-AnI4Vtugw9afFeAQd6uwH4w_TXZZWzkolgttN3SzINRmFj2Llova1flaJE6QxRLHYqleLqQy9RlavcuZE$ . You are receiving this because you modified the open/close state.Message ID: @.***>

[kubu4]

Sounds good. I'll try to get @mattgeorgephd files rsync'd today. Will plan on deleting tomorrow.

[sr320]

@kubu4 what's the general space availability on all disk one, two and three? Feel like two is our biggest problem. I would like to suggest that I come up with a list of all users/directories that I think are done on disk 2. We move those to gannet - eventually delete.

[kubu4]

Alrighty, @mattgeorgephd's files/folders have been moved to /home/shared/8TB_HDD_03/mattgeorgephd.

@meganewing - You should be able to continue your analysis; sorry for the bump in the road. However, it would be good to add a step to your script to remove each SAM file after you've created its corresponding BAM file...

That leaves the drives capacities looking like this:

I'll post a list of all users on /home/shared/8TB_HDD_03 tomorrow.

[kubu4]

Here's the full list of users on HDD_02, and how much space they're using:

1.1T    ./graceac9
838G    ./mewing0
596G    ./shedurkin
388G    ./zbengt
287G    ./bzhang11
254G    ./graceleuchtenberger
253G    ./cnmntgna
202G    ./schulh2
202G    ./pver2
167G    ./stanja
137G    ./sarahyerrace
103G    ./strigg
95G ./skalleyc
93G ./kryst21
84G ./jfaraya
81G ./albers4
77G ./sggarcia
77G ./sdiaz26
74G ./ailenp
73G ./yaaminiv
73G ./javirodr
63G ./nbravos
58G ./jwinter2
58G ./jvello
54G ./cvaldi
47G ./sr320
47G ./dani2310
32G ./anag9
30G ./hputnam
22G ./acasuso
21G ./ashuff
7.4G    ./aspen
6.6G    ./lh3
5.9G    ./davnoor
3.3G    ./yleal
3.3G    ./bg_data
3.0G    ./josefasv
1.6G    ./ece9
1.5G    ./conysaez
688M    ./mmunozt
659M    ./noahk22
54M ./mattgeorgephd

[sr320]

Lets move the following off to an archive directory on gannet.

287G    ./bzhang11
    202G    ./schulh2
    202G    ./pver2
    137G    ./sarahyerrace
    95G ./skalleyc
    93G ./kryst21
    84G ./jfaraya
    81G ./albers4
    77G ./sggarcia
    77G ./sdiaz26
    74G ./ailenp
    73G ./javirodr
    63G ./nbravos
    58G ./jwinter2
    58G ./jvello
    47G ./dani2310
    32G ./anag9
    30G ./hputnam
    22G ./acasuso
    5.9G    ./davnoor
    3.3G    ./yleal
    3.3G    ./bg_data
    3.0G    ./josefasv
    1.6G    ./ece9
    1.5G    ./conysaez
    688M    ./mmunozt
    659M    ./noahk22

[sr320]

if not too late, need to keep

5.9G ./davnoor

[kubu4]

Will do!

[kubu4]

Arlighty, accounts have been archived to Gannet and things are looking better:

I'll close this for now.

[sr320]

@kubu4 maybe you did not delete these directories after copying to gannet? https://d.pr/i/fERM4n

[kubu4]

I didn't actually delete the full directories - figured it would create a headache if someone wanted to log back in. I did delete most stuff in the directories.

However, I'm not seeing what you're seeing in that screenshot (specifically directory sizes). Here's output from that command I ran earlier:

[sr320]

Gotcha

Those dir sizes were just from issue itself before you clean up

On Tue, Jun 11, 2024 at 4:50 PM kubu4 @.***> wrote:

I didn't actually delete the full directories - figured it would create a headache if someone wanted to log back in. I did delete most stuff in the directories.

However, I'm not seeing what you're seeing in that screenshot (specifically directory sizes). Here's output from that command I ran earlier:

image.png (view on web) https://urldefense.com/v3/__https://github.com/RobertsLab/resources/assets/4514104/f85569b6-41b4-4eda-90c9-612f91b9184d__;!!K-Hz7m0Vt54!g6ITlT-MIPWCq4w1XI3UJZ7J6Uc1HZMI2iqyiJpg8l_QobaFxZ8R_pKQ4ncxxhrSz1sUzyZB5c1qxb7jvY6THUo$

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/RobertsLab/resources/issues/1677*issuecomment-2161792789__;Iw!!K-Hz7m0Vt54!g6ITlT-MIPWCq4w1XI3UJZ7J6Uc1HZMI2iqyiJpg8l_QobaFxZ8R_pKQ4ncxxhrSz1sUzyZB5c1qxb7je0MyMew$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABB4PNYZCLM7L6BJOAWZYA3ZG6ELFAVCNFSM6AAAAAA2JCNHF2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRRG44TENZYHE__;!!K-Hz7m0Vt54!g6ITlT-MIPWCq4w1XI3UJZ7J6Uc1HZMI2iqyiJpg8l_QobaFxZ8R_pKQ4ncxxhrSz1sUzyZB5c1qxb7jMvW5fN4$ . You are receiving this because you modified the open/close state.Message ID: @.***>

[kubu4]

@shedurkin - This was reopened based on your Slack message yesterday. Are you still having an issue? If not, what step(s) did you take to resolve things?

[shedurkin]

I deleted some old bam files from a previous Hisat run to clear up space and was able to finish converting everything

[kubu4]

Raven HDD02 is currently out of space:

@grace-ac has indicated she'll delete stuff, as she's using the most space ATM.

[grace-ac]

@kubu4 I just ran the code you did above to see how much space i've cleared up since i deleted a bunch of files... and it said i'm not one of the sudo-ers... and that the incident will be reported... reported to you?

[kubu4]

Ha! No report gets sent. Probably gets logged somewhere or something. I'll check out the available space and report back soon.

[sr320]

No report gets sent. Probably gets logged somewhere or something

I don't know... I just got a call.. they were pretty upset.

[kubu4]

Looks good for now. Thanks! That should allow you (and others) to continue computing. However, please feel free to remove additional files when you have the chance.