jpmckinney
commented
2 months ago I just committed a change to configure these with environment variable, e.g.
env SCRAPYD_HTTP_PORT=8080 SCRAPYD_BIND_ADDRESS=0.0.0.0 scrapydThis allows for more flexible configuration.
Do you still need CLI options?
Introduction
Scrapyd is a popular tool used for deploying and running Scrapy spiders. While Scrapyd provides various configuration options, two commonly used parameters,
--bind-addressand--http_port, require modifying the Scrapyd startup command. This article proposes enhancements to Scrapyd to add these parameters as command-line options, simplifying the configuration process.Adding
--bind-addressparameterCurrently, Scrapyd listens on the localhost (127.0.0.1) by default, limiting its accessibility. By introducing the
--bind-addressparameter, users will have the flexibility to specify the IP address on which Scrapyd should listen. This allows Scrapyd to accept connections from different hosts on a network, facilitating better integration with other services.Example usage:
In the above example, Scrapyd will bind to all available network interfaces, making it accessible from any IP address.
Adding
--http_portparameterScrapyd uses port 6800 for its HTTP API by default. However, certain environments or security configurations may require using a different port. By introducing the
--http_portparameter, users will have the ability to specify a custom port for Scrapyd's HTTP API.Example usage:
In the above example, Scrapyd will start the HTTP API on port 8080 instead of the default port 6800.
Conclusion
By adding the
--bind-addressand--http_portparameters as command-line options in Scrapyd, users will have more flexibility in configuring the network interface and port for Scrapyd's operation. This enhancement simplifies the process of modifying these commonly used parameters, making it more convenient for users to customize Scrapyd's behavior to their specific requirements.