search

trailofbits / algo

Set up a personal VPN in the cloud
https://blog.trailofbits.com/2016/12/12/meet-algo-the-vpn-that-works/
GNU Affero General Public License v3.0
28.95k stars 2.32k forks source link

Unable to create EC2 instance with standard options #14588

Open aptwebapps opened 1 year ago

aptwebapps commented 1 year ago

Describe the bug

Create Algo instance on EC2 using default configs - only modified the user list.

To Reproduce

Steps to reproduce the behavior:

  1. Pulled fresh from Git
  2. Updated env via pip install -U -r requirements.txt
  3. Ran ./algo
  4. Selected EC2, chose an EC2 instance, left all defaults as is

Expected behavior

Algo configuration should launch

Full log

PLAY [localhost] ***********************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Playbook dir stat] ***************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Ensure Ansible is not being run in a world writable directory] *******************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}
[DEPRECATION WARNING]: Use 'ansible.utils.ipaddr' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in 
ansible.cfg.
[WARNING]: The value '' is not a valid IP address or network, passing this value to ipaddr filter might result in breaking change in future.

TASK [Ensure the requirements installed] ***********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set required ansible version as a fact] ******************************************************************************************************************************************************************************************
ok: [localhost] => (item=ansible==6.1.0)

TASK [Just get the list from default pip] **********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Verify Python meets Algo VPN requirements] ***************************************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Verify Ansible meets Algo VPN requirements] **************************************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}
[WARNING]: Found variable using reserved name: no_log

PLAY [Ask user for the input] **********************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************************************
ok: [localhost]
[Cloud prompt]
What provider would you like to use?
    1. DigitalOcean
    2. Amazon Lightsail
    3. Amazon EC2
    4. Microsoft Azure
    5. Google Compute Engine
    6. Hetzner Cloud
    7. Vultr
    8. Scaleway
    9. OpenStack (DreamCompute optimised)
    10. CloudStack (Exoscale optimised)
    11. Linode
    12. Install to existing Ubuntu 18.04 or 20.04 server (for more advanced users)

Enter the number of your desired provider
:
3^M
TASK [Cloud prompt] ********************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set facts based on the input] ****************************************************************************************************************************************************************************************************
ok: [localhost]
[VPN server name prompt]
Name the vpn server
[algo]
:
^M
TASK [VPN server name prompt] **********************************************************************************************************************************************************************************************************
ok: [localhost]
[Cellular On Demand prompt]
Do you want macOS/iOS clients to enable "Connect On Demand" when connected to cellular networks?
[y/N]
:
^M
TASK [Cellular On Demand prompt] *******************************************************************************************************************************************************************************************************
ok: [localhost]
[Wi-Fi On Demand prompt]
Do you want macOS/iOS clients to enable "Connect On Demand" when connected to Wi-Fi?
[y/N]
:
^M
TASK [Wi-Fi On Demand prompt] **********************************************************************************************************************************************************************************************************
ok: [localhost]
[Retain the PKI prompt]
Do you want to retain the keys (PKI)? (required to add users in the future, but less secure)
[y/N]
:
^M
TASK [Retain the PKI prompt] ***********************************************************************************************************************************************************************************************************
ok: [localhost]
[DNS adblocking prompt]
Do you want to enable DNS ad blocking on this VPN server?
[y/N]
:
^M
TASK [DNS adblocking prompt] ***********************************************************************************************************************************************************************************************************
ok: [localhost]
[SSH tunneling prompt]
Do you want each user to have their own account for SSH tunneling?
[y/N]
:
^M
TASK [SSH tunneling prompt] ************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set facts based on the input] ****************************************************************************************************************************************************************************************************
ok: [localhost]

PLAY [Provision the server] ************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************************************
ok: [localhost]

--> Please include the following block of text when reporting issues:

Algo running on: macOS 13.0
Created from git clone. Last commit: 1083b4b Update install.sh (#14575)
Python 3.9.14
Runtime variables:
    algo_provider "ec2"
    algo_ondemand_cellular "False"
    algo_ondemand_wifi "False"
    algo_ondemand_wifi_exclude "X251bGw="
    algo_dns_adblocking "False"
    algo_ssh_tunneling "False"
    wireguard_enabled "True"
    dns_encryption "True"

TASK [Display the invocation environment] **********************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Install the requirements] ********************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Generate the SSH private key] ****************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Generate the SSH public key] *****************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Copy the private SSH key to /tmp] ************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include a provisioning role] *****************************************************************************************************************************************************************************************************

TASK [cloud-ec2 : Install requirements] ************************************************************************************************************************************************************************************************
ok: [localhost]
[cloud-ec2 : pause]
Enter your AWS Access Key ID (http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html)
Note: Make sure to use an IAM user with an acceptable policy attached (see https://github.com/trailofbits/algo/blob/master/docs/deploy-from-ansible.md)
 (output is hidden):

TASK [cloud-ec2 : pause] ***************************************************************************************************************************************************************************************************************
ok: [localhost]
[cloud-ec2 : pause]
Enter your AWS Secret Access Key (http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html)
 (output is hidden):

TASK [cloud-ec2 : pause] ***************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-ec2 : set_fact] ************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-ec2 : Get regions] *********************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-ec2 : Set facts about the regions] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-ec2 : Set the default region] **********************************************************************************************************************************************************************************************
ok: [localhost]
[cloud-ec2 : pause]
What region should the server be located in?
(https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region)
    1. ap-northeast-1
    2. ap-northeast-2
    3. ap-northeast-3
    4. ap-south-1
    5. ap-southeast-1
    6. ap-southeast-2
    7. ca-central-1
    8. eu-central-1
    9. eu-north-1
    10. eu-west-1
    11. eu-west-2
    12. eu-west-3
    13. sa-east-1
    14. us-east-1
    15. us-east-2
    16. us-west-1
    17. us-west-2

Enter the number of your desired region
[14]
:
6^M
TASK [cloud-ec2 : pause] ***************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-ec2 : Set algo_region and stack_name facts] ********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-ec2 : Locate official AMI for region] **************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-ec2 : Set the ami id as a fact] ********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-ec2 : Deploy the template] *************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "events": ["StackEvent AWS::CloudFormation::Stack algo UPDATE_ROLLBACK_COMPLETE", "StackEvent AWS::CloudFormation::Stack algo UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS", "StackEvent AWS::EC2::EIP ElasticIP UPDATE_COMPLETE", "StackEvent AWS::EC2::EIP ElasticIP UPDATE_IN_PROGRESS", "StackEvent AWS::EC2::Instance EC2Instance UPDATE_COMPLETE", "StackEvent AWS::CloudFormation::Stack algo UPDATE_ROLLBACK_IN_PROGRESS", "StackEvent AWS::EC2::Instance EC2Instance UPDATE_FAILED", "StackEvent AWS::CloudFormation::Stack algo UPDATE_IN_PROGRESS"], "log": ["AWS::EC2::Instance EC2Instance UPDATE_FAILED: The instance ID 'i-0d0e6ba6174f156e2' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidInstanceID.NotFound; Request ID: 1d7caf4e-249c-47a0-8a55-c242745a542c; Proxy: null)"], "output": "Problem with UPDATE. Rollback complete", "stack_outputs": {"ElasticIP": "3.105.232.254"}, "stack_resources": [{"last_updated_time": "2023-03-13T06:47:47.899000+00:00", "logical_resource_id": "EC2Instance", "physical_resource_id": "i-0d0e6ba6174f156e2", "resource_type": "AWS::EC2::Instance", "status": "UPDATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2023-03-13T06:47:50.007000+00:00", "logical_resource_id": "ElasticIP", "physical_resource_id": "3.105.232.254", "resource_type": "AWS::EC2::EIP", "status": "UPDATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:00:59.765000+00:00", "logical_resource_id": "InstanceSecurityGroup", "physical_resource_id": "sg-02466c5f8503dc2b3", "resource_type": "AWS::EC2::SecurityGroup", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:00:35.551000+00:00", "logical_resource_id": "InternetGateway", "physical_resource_id": "igw-0b1e26a743b06232b", "resource_type": "AWS::EC2::InternetGateway", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:01:10.769000+00:00", "logical_resource_id": "Route", "physical_resource_id": "algo-Route-9I0P7LLS5BYJ", "resource_type": "AWS::EC2::Route", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:01:10.996000+00:00", "logical_resource_id": "RouteIPv6", "physical_resource_id": "algo-RouteI-1F38KQ9CJH1YF", "resource_type": "AWS::EC2::Route", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:01:15.305000+00:00", "logical_resource_id": "RouteSubnet", "physical_resource_id": "rtbassoc-0a2df9b42dfabf624", "resource_type": "AWS::EC2::SubnetRouteTableAssociation", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:00:49.566000+00:00", "logical_resource_id": "RouteTable", "physical_resource_id": "rtb-03d7d28803c64f1d3", "resource_type": "AWS::EC2::RouteTable", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:00:52.098000+00:00", "logical_resource_id": "Subnet", "physical_resource_id": "subnet-0ade576c93de8b174", "resource_type": "AWS::EC2::Subnet", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:01:29.749000+00:00", "logical_resource_id": "SubnetIPv6", "physical_resource_id": "subnet-cidr-assoc-0f145a508375c29c5", "resource_type": "AWS::EC2::SubnetCidrBlock", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:00:33.613000+00:00", "logical_resource_id": "VPC", "physical_resource_id": "vpc-038b50990cdab709e", "resource_type": "AWS::EC2::VPC", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:00:52.995000+00:00", "logical_resource_id": "VPCGatewayAttachment", "physical_resource_id": "algo-VPCGat-13OR5OQ6W0BLC", "resource_type": "AWS::EC2::VPCGatewayAttachment", "status": "CREATE_COMPLETE", "status_reason": null}, {"last_updated_time": "2021-12-06T06:00:51.524000+00:00", "logical_resource_id": "VPCIPv6", "physical_resource_id": "vpc-cidr-assoc-02e0e8cdb96821055", "resource_type": "AWS::EC2::VPCCidrBlock", "status": "CREATE_COMPLETE", "status_reason": null}]}

TASK [include_tasks] *******************************************************************************************************************************************************************************************************************
included: /Users/<user>/work/lib/algo/playbooks/rescue.yml for localhost

TASK [debug] ***************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "fail_hint": [
        "Sorry, but something went wrong!",
        "Please check the troubleshooting guide.",
        "https://trailofbits.github.io/algo/troubleshooting.html"
    ]
}

TASK [Fail the installation] ***********************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed as requested from task"}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost                  : ok=37   changed=1    unreachable=0    failed=1    skipped=5    rescued=1    ignored=0
aptwebapps commented 1 year ago

I tried with Lightsail as well, and got a different error:

"log": [
        "AWS::Lightsail::Instance Instance CREATE_FAILED: Resource handler returned message: \"The specified bundle does not exist in this region: nano_2_0 (Service: Lightsail, Status Code: 400, Request ID: 11fcdd7b-96d2-4afb-8723-2774625aa7c1)\" (RequestToken: 6b58b8f3-2efe-9d70-5dbd-7fdbda57640f, HandlerErrorCode: NotFound)"
    ]

Should I create a separate ticket for this? I wonder if it's related - in both cases I'm trying to set up the VPN in ap-southeast-2 (Sydney).

jackivanov commented 1 year ago

The instance type might not be available in selected region. You can update it in config.cfg

aptwebapps commented 1 year ago

@jackivanov Do you mean for Lightsail? I did that (changed from nano_2_0 to nano_2_2) and it worked, so my immediate need is met. The listed type for EC2 in config.cfg is t2.micro which is definitely available in ap-southeast-2 so not sure what I should change it to.

jackivanov commented 1 year ago

For EC2 I suggest you to try and remove the stacks using the console and deploy from scratch. It seem like it's trying to update some resources, not to provision new

monglebest commented 1 year ago

Hi, I am trying to deploy it on AWS lightsail. Here is the ROLLBACK_COMPLETE error that I get. May I have any idea how to solve this? I have tried to re-start the lightsail instance but it doesn't help.

TASK [cloud-lightsail : Deploy the template] ***** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the UpdateStack operation: Stack:arn:aws:cloudformation:us-west-2:xxxxxxxxx:stack/algo/4ce6d8a0-44c4-11ee-9dc6-026dfbaa8503 is in ROLLBACK_COMPLETE state and can not be updated. fatal: [localhost]: FAILED! => {"boto3_version": "1.28.35", "botocore_version": "1.31.35", "changed": false, "error": {"code": "ValidationError", "message": "Stack:arn:aws:cloudformation:us-west-2:xxxxxxxxx:stack/algo/4ce6d8a0-44c4-11ee-9dc6-026dfbaa8503 is in ROLLBACK_COMPLETE state and can not be updated.", "type": "Sender"}, "msg": "Failed to update stack algo: An error occurred (ValidationError) when calling the UpdateStack operation: Stack:arn:aws:cloudformation:us-west-2:xxxxxxxx:stack/algo/4ce6d8a0-44c4-11ee-9dc6-026dfbaa8503 is in ROLLBACK_COMPLETE state and can not be updated.", "response_metadata": {"http_headers": {"connection": "keep-alive", "content-length": "410", "content-type": "text/xml", "date": "Sun, 27 Aug 2023 10:38:40 GMT", "x-amzn-requestid": "6feb8bf9-8665-42e9-b11e-52db333f1d4d"}, "http_status_code": 400, "request_id": "6feb8bf9-8665-42e9-b11e-52db333f1d4d", "retry_attempts": 0}}

========================= I have updated the policies to below to reach this place. { "Version": "2012-10-17", "Statement": [ { "Sid": "PreDeployment", "Effect": "Allow", "Action": [ "lightsail:DescribeImage", "lightsail:DescribeKeyPairs", "lightsail:DescribeRegions", "lightsail:ImportKeyPair", "lightsail:GetRegions", "cloudformation:DescribeStacks", "cloudformation:CreateStack", "cloudformation:ListStackResources", "cloudformation:UpdateStack", "lightsail:CopyImage" ], "Resource": [ "*" ] } ] }

glennschler commented 1 year ago

@aptwebapps @monglebest One tip is to always choose a new algo server name when running ./algo . For example add this cli argument: -e server_name=algo-$(date +%Y%b%d-%H%M)

If you always use the default server name 'algo' when prompted, it will reuse a previous ec2 / lightsail stack you ran earlier, which is why the log file errors show An error occurred (ValidationError) when calling the **UpdateStack** operation. Secondly, you can always ignore the ROLLBACK portions of the error messages, since that is the error recovery steps done automatically by EC2 CloudFormation after the true failure.

As @jackivanov mentions, the best is to delete old stacks from the CloudFormation service in the AWS Web Console. Also in the CloudFormation Console, scroll down the Status column until see the resources that have status CREATE_FAILED. The Status Reason will explain the true reason for failure.

image

monglebest commented 1 year ago

Thanks, @glennschler it works for me. Just need to locate the stack in the server region under AWS, otherwise, it's not there image

After trial and error, I have my working policy setup for aws lightsail shared. { "Version": "2012-10-17", "Statement": [ { "Sid": "PreDeployment", "Effect": "Allow", "Action": [ "lightsail:ImportKeyPair", "lightsail:GetRegions", "cloudformation:DescribeStacks", "cloudformation:CreateStack", "cloudformation:ListStackResources", "cloudformation:UpdateStack", "cloudformation:DescribeStackEvents", "lightsail:GetInstance", "lightsail:CreateInstances", "lightsail:TagResource", "lightsail:DeleteInstance", "lightsail:PutInstancePublicPorts", "lightsail:GetStaticIp", "lightsail:AllocateStaticIp", "lightsail:AttachStaticIp", "lightsail:StartInstance" ], "Resource": [ "*" ] } ] }