Hayk2508 / web-scrapping

0 stars 1 forks source link

Add DB models and CRUD #26

Closed harutmarg closed 2 months ago

harutmarg commented 4 months ago

Motivation

We want to save all objects during parsing to database, in order to save a history of outputs of our parsers. Also, the data can be used in CRUD operations for some frontend apps

TODO

Hayk2508 commented 4 months ago

I want to create model named 'MediaContent' and in this model have these fields:
url = models.URLField() - this is url of the website parsed_urls = models.JSONField(default=list) - this represents parsed_urls of this website created_at = models.DateTimeField(auto_now_add=True) updated_at = models.DateTimeField(auto_now=True) Is this correct?

harutmarg commented 4 months ago

Yes, let's create an abstract model with datetime fields and inherit every model from it Also, maybe it makes sense to also create an abstract Parser model and inherit all parsers from that And I think it's more scalable to add those two image and video parser models, even if they don't add new fields at this moment

Hayk2508 commented 4 months ago

Is this correct relation? Url->Parser->ParsedObject So at the end should i represent ParsedObject?