clear space on /web-archiving-stage volume?

ndushay commented 7 years ago

We have 2 mount points for crawl data on was-robots1-prod:

was-archiving-stage
was-archiving-stage-2 -- we filled up the first one and this was set up for additional crawls we wanted to accession

Should we have some sort of plan/process to either

move crawl data to be accessioned to the first volume
get rid of accessioned crawl data to make room for new crawl data

volume is mounted on (or in puppet):

webcapture-prod
was-downloader
was-registrar
was-robots1-prod
manifests/nodes/lyberservices.pp

as of 2017-01-20:

Filesystem            Size  Used Avail Use% Mounted on
sf4-webapp:/vol/web_archiving_stage_prod
                      6.0T  5.6T  443G  93% /web-archiving-stage
sf3-webapp:/vol/web_archiving_stage_prod_02
                      7.0T  3.3T  3.8T  47% /web-archiving-stage-2

[was@was-robots1-prod ~]$ du -sh /web-archiving-stage*/jobs/*
2.6T    /web-archiving-stage-2/jobs/SUL_ag_1
705G    /web-archiving-stage-2/jobs/SUL_ag_2

769M    /web-archiving-stage/jobs/AIT_1023
164K    /web-archiving-stage/jobs/AIT_1078
4.0K    /web-archiving-stage/jobs/AIT_1114
4.0K    /web-archiving-stage/jobs/AIT_1117
16G /web-archiving-stage/jobs/AIT_1208
4.0K    /web-archiving-stage/jobs/AIT_1515
488K    /web-archiving-stage/jobs/AIT_2361
4.0K    /web-archiving-stage/jobs/AIT_2592
1.1M    /web-archiving-stage/jobs/AIT_924
4.0K    /web-archiving-stage/jobs/AIT_929
312M    /web-archiving-stage/jobs/avaa
594M    /web-archiving-stage/jobs/carter
1.2G    /web-archiving-stage/jobs/cesta
13G /web-archiving-stage/jobs/cf
3.1G    /web-archiving-stage/jobs/chinese_ngo
82M /web-archiving-stage/jobs/cidr
870M    /web-archiving-stage/jobs/digital_michelangelo
9.4G    /web-archiving-stage/jobs/edsource
24M /web-archiving-stage/jobs/fom
144M    /web-archiving-stage/jobs/French_elections
1.2G    /web-archiving-stage/jobs/fur
2.1G    /web-archiving-stage/jobs/ga
131M    /web-archiving-stage/jobs/hlc
299M    /web-archiving-stage/jobs/mandelbrot
58M /web-archiving-stage/jobs/marl
19M /web-archiving-stage/jobs/mosul_eye
1.4M    /web-archiving-stage/jobs/nsc_portfolios
217M    /web-archiving-stage/jobs/rheingold
3.9G    /web-archiving-stage/jobs/rockpile
5.4T    /web-archiving-stage/jobs/SUL_ag_1
4.0K    /web-archiving-stage/jobs/SUL_ag_2
6.6G    /web-archiving-stage/jobs/SUL_Maryam_Mirzakhani
4.0G    /web-archiving-stage/jobs/SUL_Maryam_Mirzakhani_twitter
16G /web-archiving-stage/jobs/suwebsites
40K /web-archiving-stage/jobs/template
188M    /web-archiving-stage/jobs/xcitr

nullhandle commented 7 years ago

Most of the data in SUL_ag_1 was accessioned, so I'm clearing that out now. Should allow us to move data over from /web-archiving-stage-2 and eliminate that volume sooner.

nullhandle commented 7 years ago

I've cleared off what I could and put in a ticket to have the web-archiving-stage-2 volume deleted.

sul-dlss / web-archiving

clear space on /web-archiving-stage volume? #31