search

WPRDC / wprdc-etl

MIT License
8 stars 3 forks source link

Add second wave of jobs #52

Open saylorsd opened 8 years ago

saylorsd commented 8 years ago
bsmithgall commented 8 years ago

@saylorsd I think that for this you might want to consider breaking this into two distinct projects.

  1. The pipeline-generation code, which should have a cool name and be published to pip (happy to help out with this!)
  2. The code for the actual pipelines.
saylorsd commented 8 years ago

@bsmithgall That sounds perfect! What do you think would be the best way to go about it? I was thinking I'll just make a branch where I remove the jobs and their tests and then push that to a new repo.

saylorsd commented 8 years ago

@bsmithgall Also... any ideas on a name?

bsmithgall commented 8 years ago

I'd recommend doing something like this: https://help.github.com/articles/duplicating-a-repository/

For names, I have no idea. Something about pipes?

saylorsd commented 8 years ago

@bsmithgall Since were going to keep the history anyway, would it just make sense to remove the jobs from this repo and rename it when we come up with something?

saylorsd commented 8 years ago

@bsmithgall I'm planning on removing the WPRDC-specific jobs and there tests and their respective git histories a la this method.

I copied the jobs to this repo with only their histories intact. I think I'll move the current job tests and their respective mock files to a test directory within jobs and setup a separate build checks for them.

Does this make sense?

I think this way, when developing jobs, I'll simply add wprdc-etl as a requirement for pip and refactor the tests and stuff to account for that if necessary.

bsmithgall commented 8 years ago

Yeah. Ultimately we'll need to get the other thing installed up to pypi. I am out on vacation next week, but the week after that I can probably dedicate some cycles.

saylorsd commented 8 years ago

@bsmithgall Sounds good! That'll give us some time to come up with a name for it. Have a good time!