pulumi script to spinup and setup any AWS VM #139

Maveric-k07 commented 1 month ago

Add Pulumi script to automate the deployment of DiceDB on AWS.

How to Run the Script

Configure AWS credentials:
```
aws configure
```

Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

Install requirements:
```
pip install -r requirements.txt
```

Create a new Pulumi stack or use an existing one:

pulumi stack init my-stack  # Create new stack
# OR
pulumi stack select dev     # Use existing dev stack

Review and update inputs in the Pulumi config file (Pulumi.<stack-name>.yaml)
Preview the changes:
```
pulumi preview
```
Deploy the infrastructure:
```
pulumi up
```

Implementation Details

Uses Pulumi with Python for infrastructure as code
Creates a new VPC, subnet, internet gateway, and route table
Provisions an EC2 instance with Amazon Linux 2
Installs Go, and sets up DiceDB as a systemd service

Testing Results

The deployment successfully creates the infrastructure and installs DiceDB. However, there's a connectivity issue:

The first connection and operations (SET/GET) work as expected
Subsequent attempts result in connection timeouts
The systemctl status dicedb shows the service as active and running

test script:

import redis
import logging
import time

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

REDIS_HOST = 'public ip of the ec2'
REDIS_PORT = 7379
REDIS_DB = 0
MAX_RETRIES = 3
RETRY_DELAY = 2  # seconds

def connect_to_redis():
    for attempt in range(MAX_RETRIES):
        try:
            logger.info(f"Attempting to connect to Redis (Attempt {attempt + 1}/{MAX_RETRIES})")
            r = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB, socket_timeout=10)
            r.ping()  # Test the connection
            logger.info("Successfully connected to Redis")
            return r
        except redis.exceptions.ConnectionError as e:
            logger.error(f"Failed to connect to Redis: {e}")
            if attempt < MAX_RETRIES - 1:
                logger.info(f"Retrying in {RETRY_DELAY} seconds...")
                time.sleep(RETRY_DELAY)
            else:
                logger.error("Max retries reached. Unable to connect to Redis.")
                raise

try:
      r = connect_to_redis()

    logger.info("Setting 'foo' to 'bar'")
    r.set('foo', 'bar')

    logger.info("Getting value of 'foo'")
    value = r.get('foo')
    logger.info(f"Value of 'foo': {value.decode('utf-8')}")

except redis.exceptions.RedisError as e:
    logger.error(f"Redis error occurred: {e}")
except Exception as e:
    logger.error(f"An unexpected error occurred: {e}")
finally:
    if 'r' in locals():
        logger.info("Closing Redis connection")
        r.close()

Test Script Output

# First attempt (successful)
2024-07-15 18:46:00,530 INFO - Attempting to connect to Redis (Attempt 1/3)
2024-07-15 18:46:01,362 INFO - Successfully connected to Redis
2024-07-15 18:46:01,362 INFO - Setting 'foo' to 'bar'
2024-07-15 18:46:01,570 - INFO Getting value of 'foo'
2024-07-15 18:46:01,777 INFO Value of 'foo': bar
2024-07-15 18:46:01,779 INFO Closing Redis connection

# Second attempt (timeout)
2024-07-15 18:46:03,684 INFO - Attempting to connect to Redis (Attempt 1/3)
2024-07-15 18:46:13,898 ERROR - Redis error occurred: Timeout reading from socket

I'd appreciate any insights or suggestions on addressing the connection timeout issue, as well as any improvements to the infrastructure setup or DiceDB configuration.

Note on S3 Backend

This Pulumi project uses an S3 backend for state management. Important: Ensure that the specified S3 bucket (dice-pulumi/any suitable bucketname) is created before running the Pulumi script. Alternatively we can use the pulumi cloud for storing the state.

arpitbbhayani commented 1 month ago

@Maveric-k07 Thanks for taking this up. The code looks good in the first skim. Can you add the steps that you mentioned in this issue to run the script in the docs folder as the file setting-up-in-aws.md file?

Maveric-k07 commented 1 month ago

@arpitbbhayani I have created a new docs folder and added the setting-up-in-aws.md. Please take a look. I have also talked about an issue I am facing with the connection being timed-out after the first test; can you also take a look at that.

Maveric-k07 commented 1 month ago

When you create a stack it will prompt you to enter a passphrase as follows: Enter your passphrase to unlock config/secrets (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember); you can simply hit enter if you dont want to skip remembering a passphrase.

Pulumi.dev.yaml is the config example for a stack called "dev" that i have created which has the you can use when you create a new one.

The private_key for ssh-ing into the instance will be stored in AWS parameter store

DiceDB / dice