search

TUM-DAML / seml

SEML: Slurm Experiment Management Library
Other
165 stars 29 forks source link

adding import/export utilities #120

Closed yascho closed 4 months ago

yascho commented 1 year ago

Hi!

What does this implement?

Basic functionality to import/export collections:

seml my_collection export
seml my_collection export --file my_collection.bson
seml my_collection import my_collection.bson

For example to backup collections.

Hope it helps.

yascho commented 1 year ago

Hi, global response here.

The export/import features are mainly meant for archival purposes (i.e. archiving the whole collection with all experiment data). As far as I checked the export actually includes the code too.

I do not have any use-case of merging a previously exported collection into an existing collection and there are many unclear scenarios (e.g. conflicting experiment IDs). Only thing I can think of is merging result tables, but one should do that probably rather directly with pandas (outside of seml).

Currently only BSON is supported, not sure if JSON is useful if we want to also export code etc. correctly (see [1]).

[1] https://pymongo.readthedocs.io/en/stable/api/bson/json_util.html

n-gao commented 1 year ago

Does it include the code or just the same references to the code that work for you? Since the code is generally not saved in the collection?