search

datagovuk / ckanext-dgu

CKAN extension for data.gov.uk
http://data.gov.uk/
34 stars 33 forks source link

"hulk" disk almost full #383

Closed davidread closed 8 years ago

davidread commented 8 years ago 
$ df -h /media/hulk/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1       459G  409G   50G  90% /media/hulk

Similar issue last summer: http://redmine.dguteam.org.uk/issues/1895

davidread commented 8 years ago

Looks like clean_cached_resources.py is crashing during run:

  File "/vagrant/src/ckanext-dgu/ckanext/dgu/bin/clean_cached_resources.py", line 37, in command
    else:
AttributeError: 'NoneType' object has no attribute 'state'
davidread commented 8 years ago 
        OK: 97964 total=271656883186 [(u'/media/hulk/ckan_resource_cache/03/03252396-1495-4c0a-b557-2902b67da407/house-prices-and-sales', 24568), (u'/media/hulk/c...
        Package is deleted: 13515 total=9908470038 [(u'/media/hulk/ckan_resource_cache/03/034b8965-525c-4e9c-b248-71896053906c/September11-25kpayments.pdf', 120660), (u'/media/hu...
        Resource is deleted: 11676 total=29031215176 [(u'/media/hulk/ckan_resource_cache/03/03d7200d-c321-49e0-b1b3-341267c49553/XML', 2572051), (u'/media/hulk/ckan_resource_cache...
        Not in archival table: 870 total=13933893408 [(u'/media/hulk/ckan_resource_cache/03/03e8b581-c170-4bf4-9512-5abfa1f8b958/alun%20milford.csv', 8511), (u'/media/hulk/ckan_reso...
        No matching resouce: 603 total=2605045237 [(u'/media/hulk/ckan_resource_cache/03/030ea520-3fd4-4e6d-8bae-1afb33210e59/delsecfires.csv', 55964), (u'/media/hulk/ckan_resourc...
        Time taken (h:m:s): 2:15:10.770056
Deleting 26664 files

(ckan)co@prod3 /vagrant/src/ckanext-dgu (master) $ df -h /media/hulk/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1       459G  390G   70G  85% /media/hulk

Should have deleted 9908470038+29031215176+13933893408+2605045237=55GiB. But it's only cleared 19GB (according to df).

Running the script again immediately shows only a tiny number to remove, so the script did work.

The script says the size of the cache should be 272GiB, which is not far off 268GiB shown by du:

(ckan)co@prod3 /vagrant/src/ckanext-dgu (master) $ du /media/hulk -d 1
268155728   /media/hulk/ckan_resource_cache
140471588   /media/hulk/packagezip
408627336   /media/hulk

Disk use has gone up from 65% to 85% in 6 months, which leaves about 4 months till it is full.

davidread commented 8 years ago

Adapted the clean script to clean the unneeded packagezips - ckanext/dgu/bin/clean_package_zips.py

$ sudo /home/co/ckan/bin/python ckanext/dgu/bin/clean_package_zips.py $CKAN_INI -d
        OK: 29431 total=140286472074 [(u'/media/hulk/packagezip/a612-southern-link-road.zip', 24312), (u'/media/hulk/packagezip/mobile-food-vendors-street-traders...
        Package is deleted: 2803 total=2920567158 [(u'/media/hulk/packagezip/referrals-of-red-list-discharges-to-sewers-corporate-entities.zip', 6488), (u'/media/hulk/packagezip/...
        Not in packagezip table: 473 total=410664657 [(u'/media/hulk/packagezip/core-social-housing-lettings-reason-for-housing.zip', 10732), (u'/media/hulk/packagezip/watch-forcing-d...
        Time taken (h:m:s): 0:03:36.290877
Deleting 3276 files
(ckan)co@prod3 /vagrant/src/ckanext-dgu (master) $ df -h /media/hulk/Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1       459G  387G   72G  85% /media/hulk
(ckan)co@prod3 /vagrant/src/ckanext-dgu (master) $ du /media/hulk -d 1268155728       /media/hulk/ckan_resource_cache
du: cannot read directory `/media/hulk/lost+found': Permission denied
16      /media/hulk/lost+found
137206988       /media/hulk/packagezip
405362736       /media/hulk

This helps marginally - by 3GB. Also setup the cron for it.

davidread commented 8 years ago

I also cleared temp files on the main disk, caused by the archiver not being able to move files from /tmp to hulk.

(ckan)co@prod3 /tmp () $ df -h /dev/sda2
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       277G  220G   43G  84% /