[Epic] push speed

[zelima]

Push speed is quite low, ~1Mb file needs ~2 minutes to get on the website. And this considering the fact that we are in Alpha and nothing much is pushed and queued or run in parallel

Notes:

100kb:

needed 36 sec after link displayed needed 35 sec after link displayed(person 2)

1mb:

needed 45 sec after link displayed needed 42 sec after link displayed(person 2)

Acceptance Criteria

• [ ] processing small dataset 100Kb | processing time < 10 sec
• [ ] processing medium dataset 1Mb | processing time < 30 sec
• [ ] processing big dataset 1Gb | processing time < 10 min

Tasks

• [ ] research bottlenecks
• [ ] implement solutions

[zelima]

To bump the speed we removed assembler and redis instances and running the pipelines internally in specstore

• https://github.com/datahq/specstore/pull/32
• https://github.com/datahq/planner/pull/22
• https://github.com/datahq/deploy/pull/52

[zelima]

Results after latest changes:

• 100KB - needs 40 seconds for full process
  • 5 seconds for push
  • 35 seconds from link displayed in CLI until page reload
• 1mb - needs 45 seconds totally
  • 8 seconds for push
  • 37 seconds from link displayed in CLI until page reload

[zelima]

We are moving this from milestone to milestone and not progressing quite a while. Moving backlog for now

[zelima]

Closing as FIXED. Seed is not ideal as in acceptance criteria, but we are fine for now