telekom / mltb2

Machine Learning Toolbox 2
https://telekom.github.io/mltb2/
MIT License
10 stars 4 forks source link

Add ArangoDB upload tool. #140

Closed PhilipMay closed 10 months ago

PhilipMay commented 10 months ago

Like so:

arango_data_manager = ArangoDataManager.from_config_file("./arango_it3.env")

data_for_db = []
for row in df.itertuples():
    data = row._asdict()
    data["import_meta"] = meta_data_factory()
    for k, v in data.items():
        if isinstance(v, float) and math.isnan(v):
            data[k] = None
    data_for_db.append(data)
assert len(data_for_db) == len(df)

with closing(arango_data_manager.get_arango_client()) as arango_client:
    connection = arango_data_manager.get_connection(arango_client)
    collection = connection.collection(arango_data_manager.collection_name)
    collection.import_bulk(
        data_for_db,
        halt_on_error=True,
        details=False,
        overwrite=False,
        on_duplicate="error",
        sync=True,
        batch_size=100,
    )