search

datahubio / datahub-v2-pm

Project management (issues only)
8 stars 2 forks source link

Develop /import page #90

Open anuveyatsu opened 6 years ago

anuveyatsu commented 6 years ago

As a Publisher I want to upload / store / import data / publish data on the DataHub without installing an app so that I can test it out and start publishing data

Specifically so that:

Summary:

User flow (under discussion):

Journey through site (under discussion)

Acceptance criteria

Tasks

Phase 0 - boot repo and stub (3)

Phase 1 - UI and core classes and file selector (19)

Jump to phase 4 (?) - probably not

Phase 2

Phase 3

Phase 4

Phase 5 - Integrate into frontend

Analysis

Questions and answers

Questions

Existing work and lessons from it

Existing work:

Lessons from previous efforts:

Outline of code


var pipelineDescriptor = [
  {
    // step
  }
]
var pipelineEvents = [
  {
    uuid: step id / 'global' in case of general error
    e: event type ('start', 'done', 'err', 'rs' (schema), 'r' (row), 've' (validation error)),
    idx: row index (where applicable 'r', 've' events)
    idx: row index (where applicable 'r', 've' events)
    data: row data (where applicable)
    schema: schema (where applicable: 'rs' event)
    msg: error message
    field: failed field (where applicable, 've' event)
  } 
]
class UIState / UIWizardState
class DataHubApi
class DataPackagePipelineRunner encapsulates the pipeline system

// misc extra things you want
TableViewer
StepEditor

// get a pipeline from the UI
// probably in react this is stored in redux

listenToUi(pipelineDescriptor {   
  var dpp = dataPackagePipeline()

  dpp.configure(pipeline) -> prepared-pipeline-id

  dpp.run(prepared-pipeline-id, callback)
})

---

// pipeline object
Array<StepModel>
StepModel: {
  verb: string,
  uuid: string,
  options?: any
};
verbs: 
- `source`
  - options: `url`
- `skip`
  - options: `kind` (rows/columns), `amount` (number of kind to skip)
- `headers`
  - take the current first row and treat it as the header row
- `mutate`
  - options: `kind` (schema/validate), `field` (to mutate), `options` (to set in schema)
- `filter`
  - 

dpp callback is called on:
- status (pending/started/progress/done)
- result of a step
- errors
- validation errors

callback data object

{
  uuid: step id / 'global' in case of general error
  e: event type ('start', 'done', 'err', 'rs' (schema), 'r' (row), 've' (validation error)),
  idx: row index (where applicable 'r', 've' events)
  idx: row index (where applicable 'r', 've' events)
  data: row data (where applicable)
  schema: schema (where applicable: 'rs' event)
  msg: error message
  field: failed field (where applicable, 've' event)
}

https://github.com/akariv/dpp.ui-client/blob/master/src/app/server-events.service.ts https://github.com/akariv/dpp.ui-client/blob/master/src/app/step-model.ts

Anu's old analysis

System:

Components:

Redux:

Validation:

Transforms:

zelima commented 6 years ago

@anuveyatsu is this still active? If no I suggest closing