For example, when a free plan user performs a scrape for a club already scraped with a paid plan, metadata will be overwritten. To fix this we can check the number of available post_contents to the scraping user and compare with the existing metadata. If more are available to the user, overwrite and continue downloading. If more are availble in the existing metadata, skip the post entirely.
We can also check which plan status is currently joined, or even simplify this entirely by specifying the type of metadata e.g. metadata_plan0.json.
For example, when a free plan user performs a scrape for a club already scraped with a paid plan, metadata will be overwritten. To fix this we can check the number of available
post_contentsto the scraping user and compare with the existing metadata. If more are available to the user, overwrite and continue downloading. If more are availble in the existing metadata, skip the post entirely.We can also check which plan status is currently joined, or even simplify this entirely by specifying the type of metadata e.g.
metadata_plan0.json.