[1d?] Big pushes break after 10G - bad!

dotmesh-io / dotmesh

dotmesh (dm) is like git for your data volumes (databases, files etc) in Docker and Kubernetes

https://dotmesh.com

Apache License 2.0

538 stars 29 forks source link

[1d?] Big pushes break after 10G - bad! #655

Closed alice-sowerby closed 5 years ago

alice-sowerby commented 5 years ago

Kai started testing this and it broke after 10G

See https://github.com/dotmesh-io/dotmesh/issues/603 for notes on stress-testing Dotmesh in general - which should test this kind of case.

lukemarsden commented 5 years ago

Let's test pushing large datasets around. See https://dotmesh.slack.com/archives/C7PHSRL82/p1553795024221900

alaric-dotmesh commented 5 years ago

We now have some stress tests that do 20G push/pull/peer transfers... and they seem to work fine.

So whatever's happening here must be a problem with the deployment/configuration of dotmesh in dotscience, rather than DM itself. Something to do with the networking setup, I guess? Issue https://github.com/dotmesh-io/gateway/issues/144 created to investigate.