This PR sets up database loading for inactive ingredients. A scapy item was added + respective models and serializers for Django. The scrapy pipeline was adjusted to load inactive ingredients based on a many to many relationship with product models. The pipeline will check if an ingredient exists, if not it will create one. All ingredients then get added to the product model.
Rationale
This piece was missing from the last PR which adjusted data loading.
Tests
What testing did you do?
Ran 1.5g spl download without issue
Unit test passes
Attach testing logs inside a summary block:
Scrapy logs
Django test logs
testing logs
```
{'downloader/request_bytes': 1267265,
'downloader/request_count': 4385,
'downloader/request_method_count/GET': 4385,
'downloader/response_bytes': 774569612,
'downloader/response_count': 4385,
'downloader/response_status_count/200': 4385,
'elapsed_time_seconds': 343.70631,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 9, 15, 1, 40, 34, 871965),
'item_scraped_count': 4385,
'log_count/DEBUG': 8771,
'log_count/INFO': 15,
'memusage/max': 223100928,
'memusage/startup': 68337664,
'response_received_count': 4385,
'scheduler/dequeued': 4385,
'scheduler/dequeued/memory': 4385,
'scheduler/enqueued': 4385,
'scheduler/enqueued/memory': 4385,
'start_time': datetime.datetime(2020, 9, 15, 1, 34, 51, 165655)}
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...
----------------------------------------------------------------------
Ran 3 tests in 0.003s
OK
Destroying test database for alias 'default'...
```
Explanation
This PR sets up database loading for inactive ingredients. A scapy item was added + respective models and serializers for Django. The scrapy pipeline was adjusted to load inactive ingredients based on a many to many relationship with product models. The pipeline will check if an ingredient exists, if not it will create one. All ingredients then get added to the product model.
Rationale
This piece was missing from the last PR which adjusted data loading.
Tests
testing logs
``` {'downloader/request_bytes': 1267265, 'downloader/request_count': 4385, 'downloader/request_method_count/GET': 4385, 'downloader/response_bytes': 774569612, 'downloader/response_count': 4385, 'downloader/response_status_count/200': 4385, 'elapsed_time_seconds': 343.70631, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2020, 9, 15, 1, 40, 34, 871965), 'item_scraped_count': 4385, 'log_count/DEBUG': 8771, 'log_count/INFO': 15, 'memusage/max': 223100928, 'memusage/startup': 68337664, 'response_received_count': 4385, 'scheduler/dequeued': 4385, 'scheduler/dequeued/memory': 4385, 'scheduler/enqueued': 4385, 'scheduler/enqueued/memory': 4385, 'start_time': datetime.datetime(2020, 9, 15, 1, 34, 51, 165655)} Creating test database for alias 'default'... System check identified no issues (0 silenced). ... ---------------------------------------------------------------------- Ran 3 tests in 0.003s OK Destroying test database for alias 'default'... ```