Load data from a cloud (S3 bucket)

[Michal-Kolomanski]

Scope:

• [ ] Create a celery task that will load data from a link to S3 folder
• [ ] The task should return a python object (e.g. dict) where keys identify data type (preferably names that are used in our code - settings.py) and values enable quick and easy access to specific files from that directory. Do not load whole files there. Use generators or not - if you think that those values can be used multiple times. You can store full links to those files inside the directories. Dont story only names because then later on we will be forced to pass URL further in the code
• [ ] It should expect a specific dump structure - e.g. certain directory names.
• [ ] It shouldn't expect a specific file extension - .json, .zip, ... are all fine
• [ ] Do not require all directories to be present - e.g. alone folder called "dataset" is just fine - then just return dict with:

  {
  "dataset": <values>
  }

[wiktorflorian]

Add to the scope of the issue:

• [ ] validate_access -> function that checks if access to s3 bucket is granted
• [ ] validate_folders -> function that checks if dump folder has specific structure
• [ ] validate_files -> function that checks if files have .json or .json.gz extension