LAAC-LSCP / ChildProject

Python package for the management of day-long recordings of children.
https://childproject.readthedocs.io
MIT License
13 stars 5 forks source link

Recover portions of recordings that match a certain clocktime range #264

Closed lucasgautheron closed 2 years ago

lucasgautheron commented 2 years ago

Is your feature request related to a problem? Please describe.

Usually we are only interested in daytime and some recordings may contain a lot of night time. There should be a way to recover the timestamps of the recordings that match a (clock_start_time, clock_end_time) range specified by the user.

Describe the solution you'd like

We should discuss the alternatives here and try to see the bigger picture (e.g. sometimes, we don't want to throw away some of the data, but we can to aggregate statistics e.g. hour by hour, and maybe we can find a way to approach these two tasks similarly)

alecristia commented 2 years ago

In the Vanuatu dataset, some children have recordings that start in the night time. For instance, 1_2016-09-26 starts at 20:30. Often, this happens because the recording is long and it got re-started (for tech reasons or whatever). That same child has a second session starting about 7h after the first one, at about 6:30am, so in a way one could think they belong to the same session and/or that the first one should be disregarded when estimating day-time vocal activity.

We can go about this in at least three ways:

  1. Calculate metrics in a time range

For instance, we may want to only look at 6am to 6pm. So for the example child mentioned above, the first session would be ignored and the second included. Another child may have a session that starts at eg 2pm and ends at 10pm -- then only the section of the audio corresponding to 2-6pm would be considered.

Note that "for all the recordings that are concatenations of the original clips, start_time_accuracy = hour (because we are not sure there is no gap within the recordings, so we cannot infer clocktime at every point) for the others, start_time_accuracy = minute until one audio is shorter than the usual 4 hours due to the USB splitting the audio); then, start_time_accuracy = hour for all consecutive audios within that session following any audio shorter than 4 hours".

In some datasets (Tsi 2017) we don't have start times at all, but only whether the rec was launched in the AM or PM.

  1. Filter recordings based on start time only

This would work including in datasets like Tsi, but it's (a) a waste of data, since some recs started at eg 2am may nonetheless contain useful info (from 6am on); and (b) noisy, since some recs started at eg 5pm are essentially night-time ones

  1. Leave metrics at the session level, then use weights/other procedure in stats to account for variance.

For now, in Vanuatu, I'll do the last one.

But option 1 does seem the best!

alecristia commented 2 years ago

a related feature is to get metrics at the hourly level - that would also help look eg at habituation effects, time of day effects

lucasgautheron commented 2 years ago

a related feature is to get metrics at the hourly level - that would also help look eg at habituation effects, time of day effects

So, maybe an additional --time-grouper parameter to https://childproject.readthedocs.io/en/latest/metrics.html similar to https://pandas.pydata.org/docs/reference/api/pandas.Grouper.html ?

lucasgautheron commented 2 years ago

Problems to think of:

alecristia commented 2 years ago

not totally sure I follow the second one, but looking forward to discussing this!

alecristia commented 2 years ago

I'm trying to test the CLI version and I'm getting a child-project: error: unrecognized arguments: --from 09:00 --to 18:00. I've already: pip3 install git+https://github.com/LAAC-LSCP/ChildProject.git --force-reinstall and pip3 uninstall ChildProject and then pip3 install ChildProject --upgrade

It's unclear to me that I succeed or not because it tells me Successfully installed ChildProject-0.0.1 -- any reason why we don't change version numbers?

lucasgautheron commented 2 years ago

It's either --from-time or -f (and --to-time or -t) (see https://github.com/LAAC-LSCP/ChildProject/blob/master/ChildProject/pipelines/metrics.py#L517) does this work this way?

alecristia commented 2 years ago

what a bad user, who doesn't read the docs properly! Sorry about that. But I still can't get it to work: (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ child-project metrics --by session_id input/vanuatu/ vanuatu.csv aclew --f 09:00 --t 18:00 usage: child-project metrics path destination aclew [-h] [--vtc VTC] [--alice ALICE] [--vcm VCM] [--threads THREADS] child-project metrics path destination aclew: error: argument --threads: invalid int value: '18:00' (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ child-project metrics --by session_id input/vanuatu/ vanuatu.csv aclew --from-time 09:00 --to-time 18:00 usage: child-project [-h] {validate,import-annotations,merge-annotations,intersect-annotations,remove-annotations,rename-annotations,import-data,overview,compute-durations,process,sampler,zooniverse,eaf-builder,anonymize,metrics} ... child-project: error: unrecognized arguments: --from-time 09:00 --to-time 18:00 (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ child-project metrics --by session_id input/vanuatu/ vanuatu.csv aclew -from-time 09:00 -to-time 18:00 usage: child-project [-h] {validate,import-annotations,merge-annotations,intersect-annotations,remove-annotations,rename-annotations,import-data,overview,compute-durations,process,sampler,zooniverse,eaf-builder,anonymize,metrics} ... child-project: error: unrecognized arguments: -from-time 09:00 -to-time 18:00 (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ child-project metrics --by session_id input/vanuatu/ vanuatu.csv aclew -f 09:00 -t 18:00 usage: child-project [-h] {validate,import-annotations,merge-annotations,intersect-annotations,remove-annotations,rename-annotations,import-data,overview,compute-durations,process,sampler,zooniverse,eaf-builder,anonymize,metrics} ... child-project: error: unrecognized arguments: -f 09:00 -t 18:00 (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ child-project metrics --by session_id input/vanuatu/ vanuatu.csv aclew --f 09:00 --t 6pm usage: child-project metrics path destination aclew [-h] [--vtc VTC] [--alice ALICE] [--vcm VCM] [--threads THREADS] child-project metrics path destination aclew: error: argument --threads: invalid int value: '6pm'

lucasgautheron commented 2 years ago

I think normal users don't instantly digest the whole information provided within the docs!

Also the reason why this is failing is due to argparse being a little tricky;

since --from-time and --to-time apply to the metrics subcommand, they should appear before the aclew subcommand:

child-project metrics --by session_id input/vanuatu/ vanuatu.csv --f 09:00 --t 18:00 aclew

Hope this works

alecristia commented 2 years ago

(ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ child-project metrics -f 09:00 -t 18:00 --by session_id input/vanuatu/ vanuatu.csv aclew usage: child-project metrics [-h] [--recordings RECORDINGS] [--by {recording_filename,session_id,child_id}] path destination {lena,aclew} ... child-project metrics: error: argument pipeline: invalid choice: 'input/vanuatu/' (choose from 'lena', 'aclew') (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ child-project metrics --by session_id input/vanuatu/ vanuatu.csv -f 09:00 -t 18:00 aclew usage: child-project metrics [-h] [--recordings RECORDINGS] [--by {recording_filename,session_id,child_id}] path destination {lena,aclew} ... child-project metrics: error: argument pipeline: invalid choice: '09:00' (choose from 'lena', 'aclew')

lucasgautheron commented 2 years ago

This worked for me on oberon, using HEAD of master:

srun -c 32 child-project metrics -f 09:00 -t 18:00 --by session_id input/vanuatu/ vanuatu.csv aclew --threads 32

(the srun -c 32 part is just to use slurm on the cluster with 32 cpus, but you can remove that part)

alecristia commented 2 years ago

(ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ srun -c 32 child-project metrics -f 09:00 -t 18:00 --by session_id input/vanuatu/ vanuatu.csv aclew --threads 32 srun: job 175533 queued and waiting for resources srun: job 175533 has been allocated resources usage: child-project metrics [-h] [--recordings RECORDINGS] [--by {recording_filename,session_id,child_id}] path destination {lena,aclew} ... child-project metrics: error: argument pipeline: invalid choice: 'input/vanuatu/' (choose from 'lena', 'aclew') srun: error: puck2: task 0: Exited with exit code

now that is just weird...

lucasgautheron commented 2 years ago

Your package seems out of date still

Also, I am not bumping version numbers until right before a new release; the reason for it is that I want pip install ChildProject (i.e. the installation of ChildProject from pypi repositories rather than the HEAD of the master branch) to overwrite any previous installation if it is earlier than the pypi version

alecristia commented 2 years ago

that worked! for the record, here is my output: (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ pip3 uninstall ChildProject Found existing installation: ChildProject 0.0.1 Uninstalling ChildProject-0.0.1: Would remove: /scratch1/home/acristia/ChildProjectVenv/bin/child-project /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject-0.0.1.dist-info/ /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/ Proceed (Y/n)? y Successfully uninstalled ChildProject-0.0.1 (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ pip3 install git+https://github.com/LAAC-LSCP/ChildProject.git Collecting git+https://github.com/LAAC-LSCP/ChildProject.git Cloning https://github.com/LAAC-LSCP/ChildProject.git to /tmp/pip-req-build-qmw4va3g Running command git clone -q https://github.com/LAAC-LSCP/ChildProject.git /tmp/pip-req-build-qmw4va3g Resolved https://github.com/LAAC-LSCP/ChildProject.git to commit a1c22aa540bcd4122b8998913fc63c9d94a17113 Requirement already satisfied: pandas>=0.25.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (1.1.5) Requirement already satisfied: jinja2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (3.0.1) Requirement already satisfied: numpy>=1.16.5 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (1.20.3) Requirement already satisfied: sox in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (1.4.1) Requirement already satisfied: datalad in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (0.14.7) Collecting requests==2.25.0 Downloading requests-2.25.0-py2.py3-none-any.whl (61 kB) |████████████████████████████████| 61 kB 454 kB/s Requirement already satisfied: lxml in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (4.6.3) Requirement already satisfied: pympi-ling in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (1.70.2) Requirement already satisfied: pylangacq in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (0.15.0) Requirement already satisfied: python-dateutil>=2.8.1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (2.8.2) Requirement already satisfied: librosa in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (0.8.1) Requirement already satisfied: pydub in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (0.25.1) Requirement already satisfied: pysoundfile in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (0.9.0.post1) Requirement already satisfied: nltk in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (3.6.2) Requirement already satisfied: sklearn in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (0.0) Requirement already satisfied: PyYAML in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (5.4.1) Requirement already satisfied: panoptes-client in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (1.4.0) Requirement already satisfied: importlib-resources in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (5.2.2) Requirement already satisfied: pygamma-agreement in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (0.4.0) Requirement already satisfied: pyannote.metrics in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from ChildProject==0.0.1) (3.0.1) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from requests==2.25.0->ChildProject==0.0.1) (1.25.11) Requirement already satisfied: idna<3,>=2.5 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from requests==2.25.0->ChildProject==0.0.1) (2.10) Requirement already satisfied: chardet<4,>=3.0.2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from requests==2.25.0->ChildProject==0.0.1) (3.0.4) Requirement already satisfied: certifi>=2017.4.17 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from requests==2.25.0->ChildProject==0.0.1) (2021.5.30) Requirement already satisfied: pytz>=2017.2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pandas>=0.25.0->ChildProject==0.0.1) (2021.1) Requirement already satisfied: six>=1.5 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from python-dateutil>=2.8.1->ChildProject==0.0.1) (1.16.0) Requirement already satisfied: appdirs in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (1.4.4) Requirement already satisfied: fasteners>=0.14 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (0.16.3) Requirement already satisfied: keyrings.alt in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (4.1.0) Requirement already satisfied: jsmin in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (2.2.2) Requirement already satisfied: python-gitlab in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (2.10.0) Requirement already satisfied: humanize in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (3.11.0) Requirement already satisfied: whoosh in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (2.7.4) Requirement already satisfied: patool>=1.7 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (1.12) Requirement already satisfied: simplejson in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (3.17.5) Requirement already satisfied: importlib-metadata in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (4.8.1) Requirement already satisfied: wrapt in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (1.12.1) Requirement already satisfied: msgpack in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (1.0.2) Requirement already satisfied: iso8601 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (0.1.16) Requirement already satisfied: keyring>=8.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (23.1.0) Requirement already satisfied: tqdm in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (4.62.2) Requirement already satisfied: annexremote in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (1.5.0) Requirement already satisfied: boto in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (2.49.0) Requirement already satisfied: PyGithub in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from datalad->ChildProject==0.0.1) (1.55) Requirement already satisfied: SecretStorage>=3.2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from keyring>=8.0->datalad->ChildProject==0.0.1) (3.3.1) Requirement already satisfied: jeepney>=0.4.2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from keyring>=8.0->datalad->ChildProject==0.0.1) (0.7.1) Requirement already satisfied: zipp>=0.5 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from importlib-metadata->datalad->ChildProject==0.0.1) (3.5.0) Requirement already satisfied: typing-extensions>=3.6.4 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from importlib-metadata->datalad->ChildProject==0.0.1) (3.10.0.1) Requirement already satisfied: cryptography>=2.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from SecretStorage>=3.2->keyring>=8.0->datalad->ChildProject==0.0.1) (3.4.8) Requirement already satisfied: cffi>=1.12 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from cryptography>=2.0->SecretStorage>=3.2->keyring>=8.0->datalad->ChildProject==0.0.1) (1.14.6) Requirement already satisfied: pycparser in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from cffi>=1.12->cryptography>=2.0->SecretStorage>=3.2->keyring>=8.0->datalad->ChildProject==0.0.1) (2.20) Requirement already satisfied: future in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from annexremote->datalad->ChildProject==0.0.1) (0.18.2) Requirement already satisfied: setuptools in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from humanize->datalad->ChildProject==0.0.1) (57.4.0) Requirement already satisfied: MarkupSafe>=2.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from jinja2->ChildProject==0.0.1) (2.0.1) Requirement already satisfied: decorator>=3.0.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (5.0.9) Requirement already satisfied: joblib>=0.14 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (1.0.1) Requirement already satisfied: scikit-learn!=0.19.0,>=0.14.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (0.24.2) Requirement already satisfied: resampy>=0.2.2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (0.2.2) Requirement already satisfied: numba>=0.43.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (0.54.0) Requirement already satisfied: scipy>=1.0.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (1.7.1) Requirement already satisfied: packaging>=20.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (21.0) Requirement already satisfied: audioread>=2.0.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (2.1.9) Requirement already satisfied: soundfile>=0.10.2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (0.10.3.post1) Requirement already satisfied: pooch>=1.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from librosa->ChildProject==0.0.1) (1.5.1) Requirement already satisfied: llvmlite<0.38,>=0.37.0rc1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from numba>=0.43.0->librosa->ChildProject==0.0.1) (0.37.0) Requirement already satisfied: pyparsing>=2.0.2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from packaging>=20.0->librosa->ChildProject==0.0.1) (2.4.7) Requirement already satisfied: threadpoolctl>=2.0.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from scikit-learn!=0.19.0,>=0.14.0->librosa->ChildProject==0.0.1) (2.2.0) Requirement already satisfied: regex in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from nltk->ChildProject==0.0.1) (2021.8.28) Requirement already satisfied: click in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from nltk->ChildProject==0.0.1) (7.1.2) Requirement already satisfied: python-magic<0.5,>=0.4 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from panoptes-client->ChildProject==0.0.1) (0.4.24) Requirement already satisfied: redo>=1.7 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from panoptes-client->ChildProject==0.0.1) (2.0.4) Requirement already satisfied: tabulate>=0.7.7 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pyannote.metrics->ChildProject==0.0.1) (0.8.9) Requirement already satisfied: pyannote.core>=4.1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pyannote.metrics->ChildProject==0.0.1) (4.1) Requirement already satisfied: matplotlib>=2.0.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pyannote.metrics->ChildProject==0.0.1) (3.4.3) Requirement already satisfied: pyannote.database>=4.0.1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pyannote.metrics->ChildProject==0.0.1) (4.1.1) Requirement already satisfied: docopt>=0.6.2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pyannote.metrics->ChildProject==0.0.1) (0.6.2) Requirement already satisfied: sympy>=1.1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pyannote.metrics->ChildProject==0.0.1) (1.8) Requirement already satisfied: cycler>=0.10 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from matplotlib>=2.0.0->pyannote.metrics->ChildProject==0.0.1) (0.10.0) Requirement already satisfied: pillow>=6.2.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from matplotlib>=2.0.0->pyannote.metrics->ChildProject==0.0.1) (8.3.1) Requirement already satisfied: kiwisolver>=1.0.1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from matplotlib>=2.0.0->pyannote.metrics->ChildProject==0.0.1) (1.3.2) Requirement already satisfied: sortedcontainers>=2.0.4 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pyannote.core>=4.1->pyannote.metrics->ChildProject==0.0.1) (2.4.0) Requirement already satisfied: typer[all]>=0.2.1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pyannote.database>=4.0.1->pyannote.metrics->ChildProject==0.0.1) (0.3.2) Requirement already satisfied: mpmath>=0.19 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from sympy>=1.1->pyannote.metrics->ChildProject==0.0.1) (1.2.1) Requirement already satisfied: colorama<0.5.0,>=0.4.3 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from typer[all]>=0.2.1->pyannote.database>=4.0.1->pyannote.metrics->ChildProject==0.0.1) (0.4.4) Requirement already satisfied: shellingham<2.0.0,>=1.3.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from typer[all]>=0.2.1->pyannote.database>=4.0.1->pyannote.metrics->ChildProject==0.0.1) (1.4.0) Requirement already satisfied: cvxopt>=1.2.6 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pygamma-agreement->ChildProject==0.0.1) (1.2.6) Requirement already satisfied: cvxpy>=1.0.25 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pygamma-agreement->ChildProject==0.0.1) (1.1.15) Requirement already satisfied: TextGrid>=1.5 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from pygamma-agreement->ChildProject==0.0.1) (1.5) Requirement already satisfied: ecos>=2 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from cvxpy>=1.0.25->pygamma-agreement->ChildProject==0.0.1) (2.0.7.post1) Requirement already satisfied: scs>=1.1.6 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from cvxpy>=1.0.25->pygamma-agreement->ChildProject==0.0.1) (2.1.4) Requirement already satisfied: osqp>=0.4.1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from cvxpy>=1.0.25->pygamma-agreement->ChildProject==0.0.1) (0.6.2.post0) Requirement already satisfied: qdldl in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from osqp>=0.4.1->cvxpy>=1.0.25->pygamma-agreement->ChildProject==0.0.1) (0.1.5.post0) Requirement already satisfied: pynacl>=1.4.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from PyGithub->datalad->ChildProject==0.0.1) (1.4.0) Requirement already satisfied: pyjwt>=2.0 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from PyGithub->datalad->ChildProject==0.0.1) (2.1.0) Requirement already satisfied: deprecated in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from PyGithub->datalad->ChildProject==0.0.1) (1.2.12) Requirement already satisfied: requests-toolbelt>=0.9.1 in /scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages (from python-gitlab->datalad->ChildProject==0.0.1) (0.9.1) Using legacy 'setup.py install' for ChildProject, since package 'wheel' is not installed. Installing collected packages: requests, ChildProject Attempting uninstall: requests Found existing installation: requests 2.24.0 Uninstalling requests-2.24.0: Successfully uninstalled requests-2.24.0 Running setup.py install for ChildProject ... done

alecristia commented 2 years ago

actually, I'm not sure:

(ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ srun -c 32 child-project metrics -f 09:00 -t 18:00 --by session_id input/vanuatu/ vanuatu.csv aclew --threads 32 The ALICE set ('alice') was not found in the index. The VCM set ('vcm') was not found in the index. (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ datalad status modified: input/vanuatu (dataset) (ChildProjectVenv) [acristia@oberon vanuatu_metrics]$ git status On branch master Your branch is up to date with 'origin/master'.

Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git checkout -- ..." to discard changes in working directory) (commit or discard the untracked or modified content in submodules)

modified:   input/vanuatu (untracked content)

no changes added to commit (use "git add" and/or "git commit -a")

Strange that nothing else has been updated, isn't it? Particularly vanuatu.csv...

lucasgautheron commented 2 years ago

This worked for me, but maybe you need to datalad unlock vanuatu.csv in the first place?

$ ls -l
total 28
-rw-r--r-- 1 lgautheron bootphon     0 Aug 17 10:15 CHANGELOG.md
-rw-r--r-- 1 lgautheron bootphon   540 Aug 17 10:15 Makefile
-rw-r--r-- 1 lgautheron bootphon  5555 Aug 17 10:15 README.md
drwxr-xr-x 2 lgautheron bootphon    57 Aug 17 10:15 code
drwxr-xr-x 3 lgautheron bootphon    29 Aug 17 10:19 input
-rw-r--r-- 1 lgautheron bootphon 14646 Aug 30 15:08 vanuatu.csv
(base) [lgautheron@oberon vanuatu_metrics]$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   vanuatu.csv

no changes added to commit (use "git add" and/or "git commit -a")

EDIT: actually, I am not sure this changed anything compared to the previous version indeed, let me check!

lucasgautheron commented 2 years ago

Ok, there's an issue in the code ; on it

lucasgautheron commented 2 years ago

Alright, my bad -- should be fixed! (please upgrade the package again)