Use environment variables

[VikashKothary]

Why

Configuration using environment variables is very popular. This is especially useful when running an application in Docker.

This isn't a major issue since we can technically use environment variables to configure Scrapyrt. But personally I think it would be cool to have it out-of-the-box.

Worse-case this will document how I did it and might help someone else who wanted to do this.

How

This is how I do it currently. I run:

$ scrapyrt -i ${SCRAPYRT_HOST} -p ${SCRAPYRT_PORT} -S scrapyrt.settings

where scrapyrt.settings points to my scrapy spider's settings file:

# file: scrapyrt/settings.py

DEBUG = os.getenv('DEBUG', False)

SERVICE_ROOT = os.getenv('SCRAPYRT_SERVICE_ROOT', 'scrapyrt.resources.RealtimeApi')
CRAWL_MANAGER = os.getenv('SCRAPYRT_CRAWL_MANAGER', 'scrapyrt.core.CrawlManager')
LOG_DIR = os.getenv('SCRAPYRT_LOG_DIR', 'logs')
TIMEOUT_LIMIT = os.getenv('SCRAPYRT_TIMEOUT_LIMIT', 1000)
PROJECT_SETTINGS = os.getenv('SCRAPYRT_PROJECT_SETTINGS', None)
LOG_FILE = os.getenv('SCRAPYRT_LOG_FILE', None)
LOG_ENCODING = os.getenv('SCRAPYRT_LOG_ENCODING', 'utf-8')

What

My understanding is if the above change was made to the default_settings.py then this will be available out-of-the-box. Which might be a nice feature to have.

Host and Port can potentially be set here as the defaults assuming you want CLI to override environment variables.

P.S My 2 cents is that Host and Port should also be configurable in settings.py. If you're happy to have this change, then this could be a good time to make that change.

[pawelmhm]

I'd say configuring everything via environment variables may make it a little more difficult to develop. With all settings written in settings file you just pull code and everyone can jump and start working on it. With environment variables, you have to share it with other developers somehow, you have to maintain some configuration for people. So, I'd say it is better to keep it simple, like it is now. You can always add configuration from environment in docker easily, it won't be a serious problem.

[VikashKothary]

Hi @pawelmhm, thank you for the time.

Like I mentioned, it's not a problem for me since I already use the above for a workaround and my goal was mainly to share this with other people might be interested in solving the same problem I did.

I understand if you think it's out-of-the-scope for the library but I wanted to clarify a few things if this ever comes up in the future.

1. You should use .env for development

   configuring everything via environment variables may make it a little more difficult to develop

• This feature is an enhancement which means people could still use config files if they wanted
• But also .env files are typically used for development. They list all the environment variables and are ignored from the codebase
• This is typically the same as a config file and therefore there is no impact to the complexity of development

2. Environment variables would have default

   you just pull code and everyone can jump and start working on it

• This is a fair point and I apologies for not making it clearer. My proposal is backward-compatible through the use of defaults.
• With defaults in the config, developers can pull the code and start working on it
• I've updated the ticket description to match this to give you an idea of how that would work.

3. This is not for values that you want to share

   you have to share it with other developers somehow

• I typically use environment variables for config that you way change frequently or should be secret.
• e.g If I want to change the port in my local, I can't accidentally commit my changes
• or if my crawler needed a password, I couldn't accidentally leak it.
• And any additional config that you want to share should still go in your .settings file

4. Config files and Docker don't go together

   add configuration from environment in docker

• This is the main reason I did what I did
• If you mean adding the config file to the docker image, then this isn't a best practice. It requires you to rebuild the image to change a value
• If you mean doing what I did, then it's a question why this isn't the default as you seem to use Docker too
• Specifically, it's important to note that Docker Compose supports .env out-of-the-box.

5. This is a common convention

• See the original scrapy project: https://github.com/scrapy/scrapy/blob/06f3d12c1208c380f9f1a16cb36ba2dfa3c244c5/scrapy/utils/project.py#L70
• Heroku (the only mentioned deployment platform in the wiki) wrote these guidelines: https://12factor.net/config which suggests the same.
• Scrapyd doesn't quite support it yet but there's a ticket for it open: https://github.com/scrapy/scrapyd/issues/303