[Epic] Data validation

[zelima]

As a Publisher when I push a dataset (either new or already exists) I get a page that shows me the processing status and a preview (as much as possible) straight away

... and

As a Publisher I want to push a dataset, see the error, then fix it and repush with that repush being to the same dataset i.e. without creating a new dataset (and without reuploading data! ) so that the obvious things happen (i was reuploading the same dataset!) and i don't use more of my storage

As a Publisher I want a failed push not to "break" the existing working dataset page so that my consumers don't suddenly get a broken dataset if i make a mistake

As a Consumer I want a permanent url for a given revision of a dataset so that I can always get the data for that dataset

As a Publisher I want to know if data validation failed and details of what was wrong and how to fix it so that I can quickly correct the errors

• On showcase page (?)
• Highlight (flash message?) at top - with link to where to look more
• Want summary of errors overall (which files are affected, how badly, in what rough way)
• Per file want lots of details with clarity on what to do to fix (to start with can keep it #simple).

Acceptance Criteria

• [ ] we have nice report of data validation on s3
• [ ] report is used in frontend in best way

Tasks

• [x] We want to add a new pipeline which runs goodtables validation on each source of the pipeline. https://github.com/datahq/assembler/issues/71
• [x] Specstore:
  • [x] monitor stats for each pipeline https://github.com/datahq/specstore/issues/15
  • [x] We need to store separate intermediate pipeline results independently
  • [x] Pipeline status should be pending only after all pipelines finished running (atm it becomes ‘failed’ after first failure)
  • [x] Get status/info of specific revision https://github.com/datahq/specstore/issues/17
  • [x] Get latest succesful revision id for dataset https://github.com/datahq/specstore/issues/17
  • [x] Make sure that flow statuses are correct: https://github.com/datahq/specstore/issues/18
    • [x] SUCCESS, FAIL, QUEUED, INPROGRESS
  • [x] upload API needs to return revision-id together with dataset id https://github.com/datahq/specstore/issues/19
  • [x] Use English names from planner and embed in stats/errors https://github.com/datahq/specstore/issues/20
  • [x] Collect errors for each pipeline (and modify stats collection) so that we have in the status API a new member pipelines(See example in analysis section) https://github.com/datahq/specstore/issues/21
• [x] Planner changes https://github.com/datahq/planner/issues/9
  • [x] Add English name to each processing node
• [x] Datapackage pipelines (@adam)
  • [x] Make sure that datapackage url is added to the stats object of a successful pipeline
  • [x] (should already happen, need to see why it's not working)
• [x] Bitstore changes https://github.com/datahq/bitstore/issues/20
  • [x] Check if a file already exists and return an indication for that (as part of the upload API response)
• [x] Frontend changes:
  • [x] URL scheme changes: short url logic changes - https://github.com/datahq/frontend/issues/140
  • [x] URL scheme changes: full url - https://github.com/datahq/frontend/issues/141
  • [x] When failed show which steps are failed - https://github.com/datahq/frontend/issues/142
  • [x] When in progress show which step is running - https://github.com/datahq/frontend/issues/143
  • [ ] Redirecting to short URL from full URL if success - https://github.com/datahq/frontend/issues/144
  • [x] Show validation errors for a resource - https://github.com/datahq/frontend/issues/145
• [x] CLI
  • [x] Push data will always return the full url of the revision (see below)
• [x] Electron
  • [x] After push use full URL - https://github.com/datahq/data-desktop/issues/27
• [x] Blog post

Analysis

Collect errors for pipelines Eg:

{
  "pipelines": {
      "<pipeline-id>": {
          "title": "Create a ZIP file",
          "status": "FAILED",
          "stats": null,
          "error_log": [
            "error line 1", ...
          ]
      },
      "<pipeline-id-2>": {
          "title": "Validate package contents",
          "status": "SUCCESS",
          "stats": { 
            "bytes": 1234,
            ...
          },
          "error_log": []
      }
  }
}

[zelima]

Mostly FIXED small bug in pipeline statuses. moving it as separate issue together with blog post

• https://github.com/datahq/specstore/issues/25
• https://github.com/datahq/pm/issues/71