Scraper - Deployment - Research on deployment technologies

[Adeeshaj]

Deployment - Script - EC2

Options

1. Local Machine
2. Virtual Private Server (VPS)
3. Dedicated Server - not considerd for now
4. Serverless Platforms - cannot use since limited time limits
5. Containers (Docker)
6. Web Hosting Providers
7. Cloud Services
8. Heroku
9. Web Scraping Services
10. GitHub Actions or CI/CD Services

consider the following factors:

• Cost: Different options have different pricing models, so consider your budget.
• Scalability: Ensure your chosen platform can handle your expected workload.
• Resource Constraints: Make sure the chosen environment provides the necessary resources (CPU, RAM, bandwidth).
• Monitoring and Logging: Implement proper monitoring and error handling to ensure your scraper runs smoothly.
• Terms of Service and Legal Considerations: Be aware of the terms of service of your chosen hosting provider and respect the legal aspects of web scraping.
• Data Storage: Decide where and how you'll store the scraped data.

Resources consumption: Time - 1 hour per day RAM - 1 GB CPU - 1 STORAGE - 1 GB

Option	Cost(USD)	Resource Constraints	Monitoring and Logging	TOS	DataStorage
Local Machine	0	limited bandwidth	Can	No issue	on local
EC2	0.29	Can matched	Can	No issue	on instance
DigitalOcean	6	Can matched	Can	No issue	on instance
LiNode	5	can matched	Can	No issue	on instance
GoogleCloud	3.04	can matched	can	No issue	on instance
Heroku	0.3	can matched	can	No issue	on managed database

[Adeeshaj]

Database - Aiven

Options

1. managed database
2. database on a instsance

Resources: 10 GB Storage

Option	Cost (USD)
AWS RDS	112.29
Google Cloud SQL	9.37
Microsoft Azure Database	25.82
Heroku Postgres	9
DigitalOcean Managed Databases	15
ScaleGrid	6
aiven	0
cockroach	0

from pricing choose aiven and cockroach

Aspect	Aiven for PostgreSQL	CockroachDB
Database Type	PostgreSQL	Distributed SQL
SQL Compatibility	Full compatibility	Compatibility with
	with PostgreSQL SQL	PostgreSQL SQL
Data Consistency	Strong Consistency	Strong Consistency
Use Cases	Traditional relational	Global scalability,
	databases, e-commerce	high availability,
	platforms, web apps	geographic data
		distribution
Scalability	Horizontal scaling	Horizontal scaling,
	(limited)	automatic sharding
Geographic Distribution	Single region or zone	Built for global
	(limited)	data distribution
Common Applications	E-commerce, content	Financial services,
	management systems,	IoT, e-commerce,
	traditional web apps	global-scale apps
High Availability	Options for	Built-in support
	replication and	for high
	failover	availability
Deployment Complexity	Easier for smaller	Requires additional
	scale deployments	considerations for
		distributed systems
Learning Curve	Familiar to PostgreSQL	Requires learning
	users	distributed SQL
Recommended Use Cases	Small to medium-sized	Large-scale global
	applications with	applications with
	traditional database	distributed data
	requirements	needs
Pricing Model	Pay-per-usage with	Pay-per-usage with
	various pricing tiers	various pricing
		tiers

choosing Aiven based on Database type, use cases, and Learning curve

[Adeeshaj]

pythoneverywhere.com to deploy web app