iterative / ldb

Apache License 2.0
3 stars 0 forks source link

ipc for passing datasets #498

Open jonburdo opened 2 years ago

jonburdo commented 2 years ago

We've discussed at least adding the ability to pipe logical datasets between commands with an option (e.g. --pass-ds):

ldb index s3://bucket/prefix --pass-ds | ldb add
ldb list --query something --pass-ds | ldb add -t ws:a/b/c --pass-ds | ldb instantiate -t ds1 --pass-ds

Some interprocess communication or at least a reasonable api sharing logical datasets would be useful