search

Gapminder / ddf-validation

How do you know your DDF dataset is ✅valid?
https://open-numbers.github.io/ddftools.html
GNU General Public License v3.0
7 stars 2 forks source link

validator -i function doesn't create ddfSchema array inside datapackage.json #550

Closed angiehjort closed 5 years ago

angiehjort commented 5 years ago

ddf-validation v1.19.0 node v8.15.0

clone dataset and branch from here https://github.com/geoinfobyran/ddf--sodertornsmodellen/tree/access-and-year-3-sv

delete datapackage.json run validate-ddf -i

observe that a new datapackage.json is created but there is no ddfSchema array inside it

also the content of validation log is strange loading generic content...generating ddfSchema...loading generic content...generating ddfSchema...

angiehjort commented 5 years ago

confirm a fix

angiehjort commented 5 years ago

reopening now with a different error

clone dataset and branch from here https://github.com/geoinfobyran/ddf--sodertornsmodellen/tree/access-and-year-3-sv delete datapackage.json run validate-ddf -i

✅ getting an error message that makes sense now!

edit this file ddf--sodertornsmodellen/ddf--sodertornsmodellen--src/ddf--entities--tag.csv 

tag,name,parent
geography,Geography,
education,Education,
education_by_levels,Education levels,education
employment,Employment,
population_by_4income_groups,Four income groups,income
income_by_gender,by gender,income
income,Income,
migration_by_gender,by gender,migration
migration,Migration,
migration_by_employment_by_gender,by gender,migration_by_employment
migration_by_employment,Migration — employed,
migration_by_education_by_gender,by gender,migration_by_education
migration_by_education,Migration — educated,
housing, Housing
cars, Cars
other, Other variables

put commas in the end of last 3 lines, thus making it valid but don't remove spaces before "Housing", "Cars" and "Other variables" run validate-ddf -i again

❌ getting empty error now! datapackage creation started... datapackage.json was NOT created: {}.

now remove spaces before "Housing", "Cars" and "Other variables" run validate-ddf -i again ❌ observe the process, i think root checking and resource processing happened twice. is that OK?

image

❌ see validation log, it looks weird again loading generic content...generating ddfSchema...loading generic content...generating ddfSchema...

❌ run validate-ddf now, validator crashes, but manages to report that Validation was finished successfully! WTF? 

Angies-MacBook-Pro-2:ddf--sodertornsmodellen angie$ validate-ddf
Error: Unexpected Error: column header mismatch expected: 2 columns got: 3                 ]
    at ParserStream.self.__transform (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/parser/parser_stream.js:138:48)
    at ParserStream.__parseLineData [as __handleLine] (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/parser/parser_stream.js:65:18)
    at /Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/parser/parser_stream.js:90:22
    at asyncIterator (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/extended.js:13:17)
    at Function.asyncEach (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/extended.js:31:10)
    at ParserStream.__processRows (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/parser/parser_stream.js:88:18)
    at ParserStream._parseLine [as _parse] (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/parser/parser_stream.js:190:26)
    at ParserStream._flush (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/parser/parser_stream.js:255:18)
    at ParserStream.prefinish (_stream_transform.js:137:10)
    at emitNone (events.js:106:13)

Validation was finished successfully.
/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/async/dist/async.js:903
        if (fn === null) throw new Error("Callback was already called.");
                         ^

Error: Callback was already called.
    at /Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/async/dist/async.js:903:32
    at /Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/async/dist/async.js:3835:13
    at db.fillCollection.err (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/lib/ddf-definitions/entity.js:32:17)
    at ParserStream.csv.fromStream.on.on (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/lib/data/db.js:36:13)
    at emitOne (events.js:121:20)
    at ParserStream.emit (events.js:211:7)
    at spreadArgs (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/extended.js:43:25)
    at ParserStream.emit (/Users/angie/.nvm/versions/node/v8.15.0/lib/node_modules/ddf-validation/node_modules/fast-csv/lib/parser/parser_stream.js:314:17)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
angiehjort commented 5 years ago

i retested everything ✅ ddf-validation@1.19.3