Parallelize the DB Loading

rippinrobr / baseball-stats-db

Tools to help create and maintain databases based on the Baseball Databank Files

Apache License 2.0

10 stars 2 forks source link

Parallelize the DB Loading #23

Closed rippinrobr closed 5 years ago

rippinrobr commented 6 years ago

Currently, loading the different databases is done in a serial fashion, the files are read and parsed multiple times, and the data is loaded into each db type one at a time. The new process should be

Read the files once
the data should be passed to each DB type's worker who is responsible for loading the data into the DB
Should add a verbose flag so I only print out the progress in the way I do it now when I need it

rippinrobr commented 5 years ago

This will be resolved when the new project structure is in place. This will be backups, schemes, backups and docker files with no code