search

scrapy / scrapyd-client

Command line client for Scrapyd server
BSD 3-Clause "New" or "Revised" License
770 stars 146 forks source link

expand environment variables in scrapy.cfg #105

Closed mxdev88 closed 1 year ago

mxdev88 commented 2 years ago

Most CD pipeline frameworks heavily use environnent variables to store secrets and configuration. Currently, we need to either store these secrets in scrapy.cfg and commit to the repo or substitute some config values dynamically in scrapy.cfg before calling scrapyd-deploy.

If scrapyd-client was able to expand environment variables, it would make it much easier to integrate with pipelines.

Example, we could commit below scrapy.cfg file to our repository preventing issues described above.

[deploy]
url = $SCRAPYD_URL
project = my_project
username = $SCRAPYD_USERNAME
password = $SCRAPYD_PASSWORD

And scrapyd-client commands would automatically expand environment variables.

jpmckinney commented 2 years ago

Should we add support for this $VAR syntax to scrapy.cfg, or instead just have environment variables be another way of specifying these values (at a higher priority than the file)?

mxdev88 commented 2 years ago

It would work as well by reading directly environment variables. How would we handle/define more complex cases with multiple targets? It would require a naming convention.

EDIT The reason why I propose to use scrapy.cfg config file is because it may be easier to implement by relying on ConfigParser interpolation with a custom one inspired from there.

mxdev88 commented 2 years ago

hey @jpmckinney, I've PR'ed this idea in #106. Let me know what you think. thanks!

mxdev88 commented 2 years ago

@jpmckinney do you have any more comments or suggestions on this?

jpmckinney commented 2 years ago

@mxdev88 Sorry, I haven't had much time for my maintainer role, but this is on my list.

mxdev88 commented 1 year ago

hi @jpmckinney do you think you will get a chance to review at all?

jpmckinney commented 1 year ago

Done!