WRI-CAIT - S3 - Mapping

OpenGeoScales / ogs-data-exploration

This repository contains all data exploration notebooks for OpenGeoScales project

MIT License

11 stars 0 forks source link

WRI-CAIT - S3 - Mapping #95

Open S-AI-F opened 3 years ago

S-AI-F commented 3 years ago

Once mapped json file stored in github repository is validated, you can update your mapping script by replacing input/output data with s3 buckets.

You have a specific folder by data source where you can put raw data and mapped generated data (in staging folder).

Doc:

data staging model: https://github.com/OpenGeoScales/ogs-connectors/wiki/2---Data-Model---Staging
data staging storage: https://github.com/OpenGeoScales/ogs-connectors/wiki/Datalake-AWS-S3

CJeannesson commented 3 years ago

This has been done, but without relying on a Python script.

S-AI-F commented 3 years ago

Hi @CJeannesson Can you change please the mapping script by reading raw data from S3 and writing mapped data in S3? Since, you are the first to to make this change, can you write a small tutorial of connecting python to S3 to share with the team? Here is the dedicated wiki page: https://github.com/OpenGeoScales/ogs-connectors/wiki/Datalake-AWS-S3