search

Kaggle / kaggle-api

Official Kaggle API
Apache License 2.0
6.01k stars 1.06k forks source link

kaggle competitions files not showing all the files #458

Open youyinnn opened 1 year ago

youyinnn commented 1 year ago 
kaggle competitions files -c imagenet-object-localization-challenge
name                                                         size  creationDate
----------------------------------------------------------  -----  -------------------
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000016.JPEG      176KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000007.JPEG      215KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000017.JPEG       92KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000005.JPEG       96KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000012.JPEG      204KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000014.JPEG      128KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000004.JPEG      122KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000006.JPEG      206KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000019.JPEG      113KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000018.JPEG      183KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000011.JPEG      165KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000015.JPEG      100KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000013.JPEG      102KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000008.JPEG      177KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000003.JPEG       26KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000002.JPEG      119KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000009.JPEG       10KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000020.JPEG       74KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000010.JPEG      162KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/test/ILSVRC2012_test_00000001.JPEG       33KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000009.JPEG        112KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000012.JPEG        119KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000015.JPEG        194KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000017.JPEG        193KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000018.JPEG        112KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000014.JPEG        105KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000005.JPEG        127KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000008.JPEG        105KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000002.JPEG        137KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000016.JPEG        152KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000001.JPEG        107KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000011.JPEG        257KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000004.JPEG         83KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000013.JPEG         98KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000020.JPEG        110KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000003.JPEG        120KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000007.JPEG        162KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000010.JPEG        139KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000019.JPEG         25KB  2022-09-06 15:55:41
ILSVRC/Data/CLS-LOC/val/ILSVRC2012_val_00000006.JPEG        148KB  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000009.xml   485B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000002.xml   711B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000006.xml   486B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000018.xml   486B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000010.xml   482B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000008.xml    1KB  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000004.xml   484B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000013.xml   484B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000012.xml   482B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000014.xml   484B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000019.xml   484B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000011.xml   485B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000005.xml   483B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000015.xml   485B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000020.xml   482B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000007.xml   484B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000001.xml   486B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000016.xml   484B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000003.xml   484B  2022-09-06 15:55:41
ILSVRC/Annotations/CLS-LOC/val/ILSVRC2012_val_00000017.xml    1KB  2022-09-06 15:55:41
ILSVRC/ImageSets/CLS-LOC/train_cls.txt                       40MB  2022-09-06 15:55:41
ILSVRC/ImageSets/CLS-LOC/val.txt                              1MB  2022-09-06 15:55:41
ILSVRC/ImageSets/CLS-LOC/train_loc.txt                       17MB  2022-09-06 15:55:41
ILSVRC/ImageSets/CLS-LOC/test.txt                             3MB  2022-09-06 15:55:41
LOC_train_solution.csv                                       23MB  2022-09-06 15:55:41
LOC_sample_submission.csv                                     4MB  2022-09-06 15:55:41
LOC_synset_mapping.txt                                       31KB  2022-09-06 15:55:41
LOC_val_solution.csv                                          3MB  2022-09-06 15:55:41
aimsecond commented 1 year ago

Plus one on that

jfcann commented 1 year ago

Seconding that.

I'm now going to need to complete approx 1000 sequences of 3 clicks in order to gather the data I require. (My usecase: I need to download a large selection of the competition files, but omitting a selection that total ~ 40GB that I don't have the disk space for.)

There is a --page argument (default 20) for kaggle datasets download and it seems like only twenty files from each dir above are showing. Coincidence?

jfcann commented 1 year ago

OK, for anyone that finds themselves wanting to do this, but without a solution, here is a work-around:

  1. Run a version of this kernel to walk the competition directory and push it to competition_files.csv.
    1. Change the competition to your desired competition.
    2. You have to have the competition data loaded in the script env for anything useful to come out.
  2. You can then download the competition_files.csv either manually or via the kaggle api.
dntrply commented 10 months ago

OK, for anyone that finds themselves wanting to do this, but without a solution, here is a work-around:

  1. Run a version of this kernel to walk the competition directory and push it to competition_files.csv.

    1. Change the competition to your desired competition.
    2. You have to have the competition data loaded in the script env for anything useful to come out.
  2. You can then download the competition_files.csv either manually or via the kaggle api.

An "easy" way to have the competition file content in the input directory is to launch the notebook from the competition page and then execute the code in the notebook provided by @jfcann For instance, for the competition, RSNA 2023 Abdominal Trauma Detection, navigate to the competition --> Code (https://www.kaggle.com/competitions/rsna-2023-abdominal-trauma-detection/code) --> New Notebook