search

neurodata / ndstore

code for storing neurodata images and image annotations
http://neurodata.io
Apache License 2.0
38 stars 12 forks source link

autoingest with existing datasets created via the console #393

Closed gkiar closed 7 years ago

gkiar commented 7 years ago

I created a dataset and project through the admin console, but would like to update my data via autoingest. However, despite all parameters being the same, I get an error that suggests they are not so my upload cannot be completed. If I change the dataset/project/token names, the ingest succeeds. I am using ndio==v1.1.5.

item link
code documenting this process here
info of the dataset I created in the console here
info of the dataset I created in autoingest here

As I mentioned, you can see in the above that the datasets are identical, but still autoingest is unable to proceed and states that

From ndio: "Dataset kki2009_demo already exists and is different then the chosen dataset. Please choose a different dataset name"

Thanks. G.

@jovo @perlman @Aeusman

kunallillaney commented 7 years ago

@gkiar Can you please specify which branch is this? Or which server deployment are you working on?

gkiar commented 7 years ago

Sorry, it is written within my code and info urls, so I didn't think to specify it. I'm running this on openconnecto.me

kunallillaney commented 7 years ago

Okay. Yes, this is the old autoingest process in which it creates all the project metadata(dataset, project, channel and token) for you. You need not use the project management console. If you did create the console and want to ingest more data then ensure that the channel you are ingesting is not already created. In your case, the error info might be misleading, but I think what has happened is that the project schema has changed between these 2 projects, 0.6 to 0.7. Autoingest notices that and does not allow you to create something new in and old project schema.

kunallillaney commented 7 years ago

An important point to note is that this process only works in the current branch on DSP29 and is antiquated. Nothing about this process is carried over into the new ingest process. Feel free to document this process but this is no longer active on cloud.neurodata.io. The new process utilizes config.json files which we briefly went over the other day and is based on ingest-client and required you to pre-create project metadata.

jovo commented 7 years ago

thanks kunal. the goal is to get @gkiar to be able to upload data to datascopes until the cloud is ready.

On Sat, Dec 17, 2016 at 1:25 AM, Kunal Lillaney notifications@github.com wrote:

An important point to note is that this process only works in the new branch and is antiquated. Nothing about this process is carried over into the new ingest process. Feel free to document this process but this is no longer active on cloud.neurodata.io. The new process utilizes config.json files which we briefly went over the other day and is based on ingest-client https://github.com/neurodata/ingest-client.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndstore/issues/393#issuecomment-267746333, or mute the thread https://github.com/notifications/unsubscribe-auth/AACjchEzwFbkEtjwZkxEPWBPdu1rHa1lks5rI4BVgaJpZM4LPqIJ .

-- the glass is all full: half water, half air. neurodata.io, jovo calendar https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York starting in 2017, i will be responding to emails approximately weekly, so that i can focus more on the people around me. thank you for your understanding.

kunallillaney commented 7 years ago

@jovo It think the DSP's are working with auto-ingest. @gkiar Can you try creating a new project for the same dataset. It does not work for you because the project versions are different.

jovo commented 7 years ago

@kunal - agreed, i have no reason to think it is not currently working, we just need to learn how :)

On Sat, Dec 17, 2016 at 11:08 PM, Kunal Lillaney notifications@github.com wrote:

@jovo https://github.com/jovo It think the DSP's are working with auto-ingest. @gkiar https://github.com/gkiar Can you try creating a new project for the same dataset. It does not work for you because the project versions are different.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndstore/issues/393#issuecomment-267802291, or mute the thread https://github.com/notifications/unsubscribe-auth/AACjckg5LLjGdJnvCb3R1jU-mpVd0djSks5rJLHagaJpZM4LPqIJ .

-- the glass is all full: half water, half air. neurodata.io, jovo calendar https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York starting in 2017, i will be responding to emails approximately weekly, so that i can focus more on the people around me. thank you for your understanding.

gkiar commented 7 years ago

@kunallillaney if you check the jupyter notebook I linked to in my initial post, I have updated it to try a new project and token in the same dataset, and get the same error - no new project is created.

kunallillaney commented 7 years ago

Okay. I will look into it on Monday/Tuesday.

gkiar commented 7 years ago

@kunallillaney thank you! I appreciate it :)

gkiar commented 7 years ago

Note: The link above has been updated as the file has moved. Thanks!

kunallillaney commented 7 years ago

@gkiar This has been fixed when we have ailey running on the cloud. I will slack you the new node meant currently only for Ailey data.

gkiar commented 7 years ago

Do we have any nodes that I can use for general ingest, or other people's data? I have some imunoflouresence data and more EM data that I want to ingest, as well.

-- Greg Kiar gkiar07@gmail.com

On Sat, Feb 4, 2017 at 9:31 AM, Kunal Lillaney notifications@github.com wrote:

@gkiar https://github.com/gkiar This has been fixed when we have ailey running on the cloud. I will slack you the new node meant currently only for Ailey data.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndstore/issues/393#issuecomment-277449495, or mute the thread https://github.com/notifications/unsubscribe-auth/AEqDWPBmaKa7BZd98uZyizaPxn2lvZzuks5rZIvagaJpZM4LPqIJ .

kunallillaney commented 7 years ago

@gkiar Replied in slack. Let's take this offline.