lucasgautheron
commented
3 years ago Hi, I'm trying to create a cheatsheet for myself for pushing extracting & clips onto zooniverse. I'll always do this on oberon, so I'll only think of that case.
So far I have:
datalad install git@github.com:LAAC-LSCP/solomon-data.git cd solomon-data source ~/ChildProjectVenv/bin/activate datalad run-procedure setupBut that last step fails:
I think you may have accidentally run datalad run-procedure setup foberon (probably a copy paste from the doc) right before datalad run-procedure setup (which worked, as you can see, hence why the following steps are working)
Just in case, I pushed along and the next step worked:
datalad get recordings/convertedI'm uncertain as to the following step. I think I should select segments somehow -- but in the docs the next step is the chunkification of the segments.
It is true that you need to provide segments before, which the doc currently does not clearly state. But if you do not provide these segments, you won't be able to go any further.
About chunkification, I currently have this draft of the command:
cd .. #to find myself at the same level as solomon-data, since just now I was inside solomon-data child-project zooniverse extract-chunks solomon-data --keyword talkerNtype --chunks-length 500 --segments segments.csv --destination solomon-data/annotations/zooniverse/raw --batch-size 1000Should destination be inside solomon-data or somewhere else? What happens if I leave it unspecified? Same for chunks-length & batch-size. Could/should we have a default behavior that means that the chunks will be created inside solomon-data in some place such that the actual mp3/wavs don't get included in the data but the metadata etc does?
You need to define a destination (otherwise, an error will be thrown, and the script will stop)
It is up to the user to decide where to store the output. It might not be within solomon-data; e.g., if you are developing an analysis aside, you may have imported solomon-data as a subdataset, and the chunks will preferably lie somewhere in your analysis folder. We do not expect every user to push their own chunks to the original dataset in the general case. (Also, honestly, the audio chunks do not need to be kept once they have been uploaded.)
However, in case you want to push the chunks to the dataset, I am not sure annotations is the best fitting place for that. I think a better design would be to create a 'samples' subfolder, as in here: https://github.com/LAAC-LSCP/ChildProject/issues/148#issue-816256844
You could then set the destination to solomon-data/samples/high-volubility/chunks for instance
What is batch-size, actually? Why is it declared in the chunkification stage in addition to the upload stage? I just saw in the upload stage this is optional -- shouldn't it be mandatory (or have a default of 1000) in the upload stage?
Batch size defines how many of the chunks will be grouped and uploaded together. This reproduces the behavior of Chiara's script. This is apparently needed because of Zooniverse upload rate quotas. --batch-size defines how many chunks should each batch contain. At the upload step, you need to define how many of these batches will be uploaded. This way, you can upload n batches the first day, then n more batches the second day, etc. Maybe we could avoid that, and have only one option to specify how many chunks should be uploaded during the upload - in this case, we could drop the batch system. Let me rethink this!
The step after that is chunk upload. Here is my command draft:
child-project zooniverse upload-chunks solomon-data --chunks solomon-data/annotations/zooniverse/raw/chunks.csv --project-id 14957 --set-prefix ac_20210408Have we decided on a naming convention for the prefix?
We have not. Should we ?
The next step is to create a record that I did this, but updating the data. Here is my command draft for that:
cd solomon-data datalad save annotations/zooniverse/raw -m "adding record of zoo chunks"
This is the equivalent of git add & git commit; you will also need to push the data at some point (datalad push)
Eventually, I'll get classifications:
child-project zooniverse retrieve-classifications solomon-data --project-id 14957And repeat the data update.
cd solomon-data datalad save annotations/zooniverse/raw -m "adding record of zoo chunks - annotated"
You will also have to set the destination for child-project zooniverse retrieve-classifications, e.g.:
child-project zooniverse retrieve-classifications solomon-data --destination solomon-data/samples/high-volubility/classifications_2021-04-10.csv --project-id XXXPS: you don't need to cd out of solomon-data; you could just do child-project validate . for instance
PPS: if you write a cheatsheet for Zooniverse, can you please share it ? Then I can adapt it into a tutorial for the docs.
alecristia
Hi, I'm trying to create a cheatsheet for myself for pushing extracting & clips onto zooniverse. I'll always do this on oberon, so I'll only think of that case.
So far I have:
But that last step fails:
Just in case, I pushed along and the next step worked:
I'm uncertain as to the following step. I think I should select segments somehow -- but in the docs the next step is the chunkification of the segments.
About chunkification, I currently have this draft of the command:
Should destination be inside solomon-data or somewhere else? What happens if I leave it unspecified? Same for chunks-length & batch-size. Could/should we have a default behavior that means that the chunks will be created inside solomon-data in some place such that the actual mp3/wavs don't get included in the data but the metadata etc does?
What is batch-size, actually? Why is it declared in the chunkification stage in addition to the upload stage? I just saw in the upload stage this is optional -- shouldn't it be mandatory (or have a default of 1000) in the upload stage?
The step after that is chunk upload. Here is my command draft:
Have we decided on a naming convention for the prefix?
The next step is to create a record that I did this, but updating the data. Here is my command draft for that:
Eventually, I'll get classifications:
And repeat the data update.