Separate pipeline functionality from overal ETL process

[saylorsd]

• [x] Remove config file handling from pipeline code
• [x] Move jobs to separate repo
• [ ] Move End-to-end tests to jobs repo

[saylorsd]

@bsmithgall Whenever you have a chance, let me know what you think. I did a little work changing the way config data is passed (now it's passed as kwargs instead of through reading the file) this way the settings can be stored in whatever way a user wants. I also removed the CLI methods which I'll move over to another repo as well as made status logging not the default behavior. I refactored the tests accordingly, but I still need to fix the documentation.

[bsmithgall]

I feel like it might be better to make them separate modules inside of this project instead of separate projects.

Is there an associated PR with this issue?

[saylorsd]

There's not PR for this yet, there are still some thing's I'd like to do first. I did do a little work here(5a23823332c4549376da6e34831444ae39014099) though if you want to take a look.

Is there a particular reason why we should keep them together? I was imagining it being a situation where development of the two would eventually branch off.

[bsmithgall]

I just think that the point is to compose and run jobs, but I do think that it makes sense to be able to not monitor a case and to maybe be able to separate the concerns a bit better. But, I think that it's better from an end-user perspective to be able to install only one dependnecy.

[saylorsd]

I hear ya on that. I guess what I was thinking was separating a lot of the composition and monitoring and user-facing stuff so that we could develop on that independently. Now that I think about it, I'll bring back the CLI stuff. Turning logging off by default as well as passing config parameters instead of a config file should do what I was hoping to get. I should have PR on it later today.

[saylorsd]

I'm gonna close this issue and make a new one that better represents what I'm getting at now.