search

blockchain-etl / ethereum-export-pipeline

UNMAINTAINED! AWS CloudFormation scripts for Ethereum ETL export pipeline
MIT License
8 stars 5 forks source link

Unclear definition of what a DataPipeline object is #12

Open lazzarello opened 5 years ago

lazzarello commented 5 years ago

I can't find any reason behind the CloudFormation errors I'm getting about having too many objects for a data pipeline. What is an object? How can I calculate how many of these objects are created from the Troposphere template?

majeas165 commented 5 years ago

@medvedev1088 It seems we reach AWS Data Pipeline limitation per stack. https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-limits.html

medvedev1088 commented 5 years ago

@majeas165 yes that's correct.

An object in Data Pipeline is an input, output, activity, anything else listed in this doc https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-pipeline-objects.html.

In the generated export_pipeline.template they are JSON objects in the PipelineObjects field.

One way to solve it is to increase AWS limits: read here https://medium.com/@medvedev1088/how-to-export-the-entire-ethereum-blockchain-to-csv-in-2-hours-for-10-69fef511e9a2.

Another way is to split up your templates as I explained here https://github.com/medvedev1088/ethereum-export-pipeline/issues/11