search

scrapinghub / shub

Scrapinghub Command Line Client
https://shub.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
125 stars 79 forks source link

Deployment Hooks #306

Open techdragon opened 6 years ago

techdragon commented 6 years ago

It would be useful to have pre-deployment hooks to automate simple tasks like incrementing the version number, or rotating/generating new API keys, etc... before deploying to the scrapy cloud.

jdemaeyer commented 6 years ago

Hi @techdragon,

this sounds a bit like implementing a poor man's build automation/continuous integration. Do you think there is an advantage of having this functionality in shub rather than placing these hooks in, say, a Makefile or a Travis CI config file?

techdragon commented 6 years ago

@jdemaeyer I agree it borders on poor mans automation and I was tempted to just implement it in a makefile myself. However the simple nature of the use case that prompted me to look for this feature in the source code, was just autoincrementing the version number. I think it would be overkill to bring in a larger tool or script when I'm prototyping a scraping project before I've even reached a point where the scraper justifies being in source control but incrementing that number is important for distinguishing the scraper runs on the Scrapy Cloud.

Perhaps instead of a generic deployment hook, a feature to do version autoincrementing on deployment would be a better option?

jdemaeyer commented 6 years ago

I'm prototyping a scraping project before I've even reached a point where the scraper justifies being in source control but incrementing that number is important for distinguishing the scraper runs on the Scrapy Cloud.

Hm, I'm still very reluctant to implement this in shub. If you're at a point where you want to track which version yielded which results then it sounds like there's a chance you'll want to revert to, or at least look at, an older source version at one point. To me this very much justifies revision control (even if only local).

Perhaps instead of a generic deployment hook, a feature to do version autoincrementing on deployment would be a better option?

If you absolutely want to avoid revision control, you could change setup.py to generate a time-dependent version number, e.g. version = int(time.time()) or version = datetime.datetime.now().strftime('%Y.%m.%d.%H.%M')