search

theodi / shared

Repo that we use for non-repo-specific stories and other shared stuff.
22 stars 1 forks source link

Move CSVlint file storage out of MongoDB #463

Closed Floppy closed 9 years ago

Floppy commented 9 years ago

CSVlint filled its database, and stopped working. I’ve added another shard, but that’s only a temporary solution, and I’m not sure of the cost impact if we leave it too long. We need to spend a little time making it more space and cost-efficient. We know what needs doing (moving storage to block store instead of MongoDB), but we need to spend the time to do it before too long.

quadrophobiac commented 9 years ago

will be pairing with @bcouston from Wed morning onwards on this

bcouston commented 9 years ago

Updated description: CSVLint's current functionality is that it stores uploaded files.

Originally there were two possible solutions.

  1. Move file storage to a better suited file store. Possibly cloud storage such as rackspace.
  2. Remove persistent storage of CSV files as they are only used for a few functions that only need a file to be stored temporarily.

After discussion with @pezholio we have determined solution two is the best solution.

@Floppy also suggested moving file handling into a seperate class. For tidier code and easy to track functionality. Probably under the lib directory.

quadrophobiac commented 9 years ago

Potential related bug. moved to https://github.com/theodi/csvlint/issues/195 Something to do with how dialects are formed . Edge Case where two successive clicks of Validate (and build_dialect method) will result in a badly formed URI * run server via foreman or whatever * upload fixtures/csvs/revalidate * user is presented with validation screen with message "Congratulations! Your CSV is valid!" but with a 'Structural problem warning' If 'CR-LF' "\r\n" is selected from 'Line terminator' drop down and revalidate clicked then 1 \ Structural Error is returned, however no subsequent option to revalidate is provided * if "LF\n" is selected from 'Line terminator' drop down and Revalidate clicked 2 * 'Structural problem warnings' are returned on a refreshed validation screen ~~* if Line Terminator "\r\n" selected subsequent to above step and revalidate clicked rails terminates with a Bad URI exception ~~ ~~bad URI(is not URI?): ~~

quadrophobiac commented 9 years ago
bcouston commented 9 years ago

After conversation with both Stuart and James, the decision is to keep GridFS in the application as a temporary filestore for an uploaded csv file and delete it after revalidation. This is because James stated that in production where you have more than one app server, you never know where your request will be going.

Floppy commented 9 years ago

PR submitted in theodi/csvlint#196

Floppy commented 9 years ago

Just need to work out how to run the job on a schedule.

pezholio commented 9 years ago

Got a PR for this here https://github.com/theodi/csvlint/pull/203