CSVlint is an online validation tool for CSV files. It validates conformity of CSV releases to standards, checks for missing or malformed data, and can validate against both CSVW and Datapackage schema standards.
CSVlint is a rails app designed to act as a continuous validation service, so that when data is changed online, the validation is updated. Data can be online or uploaded for private validation.
The validation code is actually all done in a Ruby gem that can also be freely reused in other projects
Follow the public feature roadmap for CSVLint
Ruby version 2.3.1
The application uses mongod and redis databases as background jobs for data persistence
.env
file (see below)
For running tests:
PUSHER_APP_ID
PUSHER_KEY
PUSHER_SECRET
PUSHER_CLUSTER
For development add:
AWS_ACCESS_KEY
AWS_BUCKET_NAME
AWS_SECRET_ACCESS_KEY
In production add:
App Keys
tab and get the values and paste them in to your .env
filePUSHER_APP_ID=
PUSHER_KEY=
PUSHER_SECRET=
NOTE: You may be set up for a non-default Pusher cluster (The default is us-east-1
), which causes some confusion. Look at your App overiew on pusher.com and get the Cluster value from the 'Keys' section. Add this to your .env
file as PUSHER_CLUSTER=
Create an AWS S3 bucket and grant its permissions accordingly
Users
Programmatic Access for Access Type
.Attach existing policies directly
- this will open a new tab in your browser.create your own policy
and give it a name, like octopub-dev-permissions
, then for the policy document, use the following template, but add your own bucket name instead of <BUCKETNAME>
.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAdminAccessToBucketOnly",
"Action": [
"s3:*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<BUCKETNAME>",
"arn:aws:s3:::<BUCKETNAME>/*"
]
}
]
}
validate policy
just to be sure you've not made a typo. Then confirm.Set permissions page
, select the policy you've just created in the table by selecting the checkbox. Then click Review
then Create user
.csv file
containing the credentials and add the following to your .env
fileAWS_ACCESS_KEY_ID=<YOURNEWUSERACCESSKEY>
AWS_SECRET_ACCESS_KEY=<YOURNEWUSERSECRET>
S3_BUCKET=<YOURNEWS3BUCKETNAME>
Install mongo:
brew install mongo redis
(if using brew)
make a data directory for mongo databases
sudo mkdir -p /data/db
change directory ownership so that mongodb can operate
sudo chown -R $USERNAME /data/
Pre-requisites: AWS account, Pusher Account - these instructions assume you have these in place already.
Checkout the repository and run bundle
in the checked out directory.
run mongo : mongod
run redis : redis-server
to initialise the app run this in root directory of app
foreman start
If you have trouble running bundle try these (if you see errors relating to openssl
, eventmachine
or therubyracer
)
brew install openssl # if not installed
gem install eventmachine -v '1.0.7' -- --with-cppflags=-I/usr/local/opt/openssl/include
gem install libv8 -v '3.16.14.13' -- --with-system-v8
gem install therubyracer -- --with-v8-dir=/usr/local/opt/v8-315
https://github.com/shakacode/react-webpack-rails-tutorial/issues/266 ~ rubyracer with CSVlint
ensure phantomjs is installed, brew install phantomjs
(or however you prefer)
To run the entire suite of rspec unit tests and cucumber features execute
bundle exec rake
alternatively execute each suite separately with
for unit tests execute bundle exec rspec
for Cucumber features execute bundle exec cucumber
rake csvs:clean_up
rake summary:generate
rake clean_up:dupes