search

jschnurr / scrapyscript

Run a Scrapy spider programmatically from a script or a Celery task - no project required.
MIT License
121 stars 26 forks source link

loading settings file #5

Closed fsecada01 closed 6 years ago

fsecada01 commented 6 years ago

Hi. I'm trying to figure out the best way to load a Settings object into the scrapyscript module. I'm trying to build this into an app within Django and the tutorial didn't point me in a direction pertaining to loading existing settings instead of the global defaults from Scrapy.

This is my current directory with the scripting going after the crawlers within crawlers.py. I'm following the example code provided in the ReadMe and I'm not sure how to proceed.

screenshot_2

fsecada01 commented 6 years ago

I wrote a function to resolve this matter by importing the dict from the settings module. Perhaps this can be something to push to git.

def get_settings(settings):
    settings_dict = settings.__dict__
    settings_list = list(settings_dict.items())[11:]
    settings_dict = dict(settings_list)
    global crawl_settings
    crawl_settings = Settings(settings_dict)
    return crawl_settings