search

scrapinghub / shub

Scrapinghub Command Line Client
https://shub.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
125 stars 79 forks source link

Normalize project representation in ShubConfig #265

Closed jdemaeyer closed 5 years ago

jdemaeyer commented 7 years ago

Currently, conf.projects can contain a mixture of integers and dictionaries. E.g., the configuration file

projects:
  default: 12345
  prod:
    id: 33333
    stack: scrapy:1.1

Would lead to 

In [2]: load_shub_config().projects
Out[2]: {'default': 12345, 'prod': {'id': 33333, 'stack': 'scrapy:1.1'}}

We should normalize all projects to dictionaries when loading configuration data. However, the syntactic sugar in scrapinghub.yml should remain.

noviluni commented 5 years ago

I've been looking into this issue and I found that you can use .normalized_projects to get that functionallity:

In [2]: load_shub_config().normalized_projects
Out[2]: {'default': {'id': 12345, 'endpoint': 'default', 'apikey': 'default'}, 'prod': {'id': 33333, 'stack': 'scrapy:1.1', 'endpoint': 'default', 'apikey': 'default'}}

Is this issue still valid?

vshlapakov commented 5 years ago

You're right, it seems it's not, thanks for checking :+1: