search

nomic-ai / nomic

Interact, analyze and structure massive text, image, embedding, audio and video datasets
https://atlas.nomic.ai
1.1k stars 150 forks source link

Drop indices on pandas dataframes #223

Open bmschmidt opened 8 months ago

bmschmidt commented 8 months ago

Adding a dataframe gives this error:

ValueError: Underscore fields are reserved for Atlas internal use: __index_level_0__

but if we just run reset_index() before uploading it'll be fine.

hvsesha commented 8 months ago

hi in which file you are getting this error Kindly specify

bmschmidt commented 8 months ago

This is an error thrown from data_inference.py in some cases when a pandas frame is uploaded.

patschw commented 3 months ago

Hi, I have the same problem with the map_data function. I am passing a numpy embedddings array and a pandas df

dataset = atlas.map_data(data=df_filtered, embeddings=df_filtered.loc[:, 'dim_0':'dim_199'].to_numpy(), id_field='word', description="A test map of some words from Weimar to BRD 1" )

bmschmidt commented 3 months ago

Thanks for the report. We'll prioritize fixing this, but I think that if you run reset_index() on your df before passing it to an Atlas method, it will hopefully resolve your issue.