MACS data - Githubissues

krishnakalyan3 commented 2 years ago

MACS data processing pipeline.

TODO:

[x] Download Script
[x] Upload to raw_data s3://s-laion-audio/raw_dataset/MACS
[x] Data Preprocess Script
[ ] Custom Split
[ ] Upload to webdataset_tar s3://s-laion-audio/webdataset_tar/MACS
[ ] Data Card

MACS dataset is an augmented version of the TAU Urban Acoustic Scenes 2019 dataset

krishnakalyan3 commented 2 years ago

cc: @marianna13 for review.

@YuchenHui22314 looks like we already have this dataset here perhaps we should deduplicate?

YuchenHui22314 commented 2 years ago

cc: @marianna13 for review.

@YuchenHui22314 looks like we already have this dataset here perhaps we should deduplicate?

Oh, it seems that they are deleted afterwards. However, you can redownload them using this script

krishnakalyan3 commented 2 years ago

@YuchenHui22314 @marianna13 could you please review every thing that is done so far?.

I am not sure about the following

Can you please point me to existing code using to split the data. This is a dataset without any splits. Where is this code checked in?
After splitting the dataset I would need to convert to webdataset_tar. The total number of pairs in this dataset is 3930. Does -num_element 512 sound good for this dataset?. Should this command below also be part of this github repo somewhere?

python make_tar.py --input /fsx/MACS/processed_datasets --output /fsx/MACS/processed_datasets/webdataset_tar/ --dataclass all --num_element 512 --filename MCAS

YuchenHui22314 commented 2 years ago

@krishnakalyan3 Please check dircord message

LAION-AI / audio-dataset

MACS data #67