search

OriHoch / data4dappl

MIT License
2 stars 7 forks source link

uploading many files at once sometimes fails #135

Open OriHoch opened 4 years ago

OriHoch commented 4 years ago

reproduction

Upload many files at once using the file uploader UI (drag & drop or select multiple files)

expected

should work

actual

some files return 500 http error when trying to download them

OriHoch commented 4 years ago

in all cases the files were actually uploaded successfully but the server was not notified about it

workaround

execute shell on one of the CKAN pods and run the following:

cd /var/lib/ckan/data/file_uploader_ui/DATASET_NAME
DIRS_NOT_STATUS_ACTIVE="$(for DIR in `ls`; do ! cat $DIR/metadata | grep '"status": "active"' >/dev/null && echo $DIR; done)"
DIRS_NOT_STATUS_ADDING="$(for DIR in $DIRS_NOT_STATUS_ACTIVE; do ! cat $DIR/metadata | grep '"status": "adding"' > /dev/null && echo $DIR; done)"
echo $DIRS_NOT_STATUS_ADDING

the DIRS_NOT_STATUS_ADDING variable should be empty, if it's not then those uploads should be investigated manually, and maybe deleted to be re-uploaded

DIRS_STATUS_ADDING="$(for DIR in `ls`; do cat $DIR/metadata | grep '"status": "adding"' > /dev/null && echo $DIR; done)"
for DIR in $DIRS_STATUS_ADDING; do ls -lah $DIR/file; done

make sure all files have logical size (depending on the upload..)

if everything seems to be in order, update status to active

for DIR in $DIRS_STATUS_ADDING; do
  sed -i 's/"status": "adding"/"status": "active"/' $DIR/metadata
done