search

uwescience / raco

Compilation and rule-based optimization framework for relational algebra. Raco is the language, optimization, and query translation layer for the Myria project.
Other
72 stars 19 forks source link

Export to S3 via MyriaL #578

Closed jortiz16 closed 6 years ago

jortiz16 commented 6 years ago

Adding the ability to export relations to S3 via MyriaL. This is not a parallel export. All workers shuffle their data to one worker before sending to S3.

By default, the relation exports in CSV format. Should we make that configurable?

Example query:

t = load("https://s3-us-west-2.amazonaws.com/uwdb/sampleData/TwitterK.csv", csv(schema(column0:int, column1:int), skip=1));
export(t, 's3://elastic-cluster-test/testExport');
coveralls commented 6 years ago

Coverage Status

Coverage decreased (-0.1%) to 91.655% when pulling a2572ca4ebe1481332d6e4f704a3271fc0cf9364 on s3Export into f57374d64b20a72a901e82a0d66a0c182b4da635 on master.