search

aws / efs-utils

Utilities for Amazon Elastic File System (EFS)
MIT License
295 stars 189 forks source link

Mounting fargate volume using EFS doesn't create sub directories #79

Open Kidist-Abraham opened 4 years ago

Kidist-Abraham commented 4 years ago

We are planing to use AWS EFS as persistent storage for our container on fargate service. The problem we are encountering is, all the directories the container uses needs to be ready prior to running the task. Or else, there will be a file not found error while the task runs. There was no sunch problem when we run the task using the non-persistent storage. Is there any work around for this?

Cappuccinuo commented 4 years ago

Hey @Kidist-Abraham ,

Thanks for reaching out to us. I will assume you are using ECS Fargate since you are talking about task.

Can you share the general idea of you task definition? Is the root cause of the issue due to the PV is not fully mounted before some directories operation?

Thanks.

Kidist-Abraham commented 4 years ago

Hey @Cappuccinuo, thanks for the response. Yes I am running fargate version 1.4.0 with AWS EFS for mounting the volume. The task Is running the image inputoutput/cardano-node:1.18.0 This image has its entry point executable in /nix/store/..../bin/entry-point. The error I am getting is (it is built with nix)

18:22:02container_linux.go:349: starting container process caused "exec: \"/nix/store/w8cwn94bw2x86rzk0qralfdl6rsnwjdw-entry-point/bin/entry-point\": stat /nix/store/w8cwn94bw2x86rzk0qralfdl6rsnwjdw-entry-point/bin/entry-point: no such file or directory"
container_linux.go:349: starting container process caused "exec: \"/nix/store/w8cwn94bw2x86rzk0qralfdl6rsnwjdw-entry-point/bin/entry-point\": no such file or directory

This error comes after the container is already started. This is the first and only log The weird thing about this is it works when I am not using AWS EFS and mount the volumes to non-persistence storage ECS provides. So I thought it has something to do with EFS

Cappuccinuo commented 4 years ago

Hey @Kidist-Abraham ,

Thanks for the explanation.

  1. Can you provide the general steps/workflows for us to reproduce this issue? It will be great if you can share the sample yaml file.

  2. Is the cardano-node this tool on github?

Thanks.

Kidist-Abraham commented 4 years ago

Yes, that is the repo.

Here is the most general step (Please let me know if something is not clear and I will provide the detail). I created the services with GUI.

  1. Create a vpc, attach subnets to it, attach network ACL to the subnets with allowing everything from everything.

  2. Create a security group ,in my case it is called cardano-wallet with the following rules on the inbound rules

image and allowing all traffic to everwhere on the outbound

  1. Create another security group to be used by EFS, in my case EFS-cardano-wallet with the following rules on the in bound rules image

and allowing all traffic to everwhere on the outbound

  1. Create an EFS files system with Performance mode - General Purpose Throughput mode - Bursting Lifecycle policy - None Automatic backups - Disabled Encrypted - No File system policy -

    {
"Version": "2012-10-17",
"Id": "ExamplePolicy01",
"Statement": [
    {
        "Sid": "ExampleSatement01",
        "Effect": "Allow",
        "Principal": {
            "AWS": "*"
        },
        "Action": [
            "elasticfilesystem:ClientRootAccess",
            "elasticfilesystem:ClientMount",
            "elasticfilesystem:ClientWrite"
        ],
        "Resource": "<FS ARN>"
    }
]
}

    Add mount target - (IP address created automatically ) image

  2. Create a task definition using the following JSON file (replace with your fs id and with your AWS account ID). I used the default ecsTaskExecutionRole and ecsTaskRole provided by AWS ECS. 

    {
"ipcMode": null,
"executionRoleArn": "arn:aws:iam::<AWS ID>:role/ecsTaskExecutionRole",
"containerDefinitions": [
{
  "dnsSearchDomains": null,
  "environmentFiles": null,
  "logConfiguration": {
    "logDriver": "awslogs",
    "secretOptions": null,
    "options": {
      "awslogs-group": "/ecs/wallet-node",
      "awslogs-region": "us-east-1",
      "awslogs-stream-prefix": "ecs"
    }
  },
  "entryPoint": null,
  "portMappings": [],
  "command": [
    "mkdir -p /nix/store"
  ],
  "linuxParameters": null,
  "cpu": 0,
  "environment": [
    {
      "name": "NETWORK",
      "value": "testnet"
    }
  ],
  "resourceRequirements": null,
  "ulimits": null,
  "dnsServers": null,
  "mountPoints": [
    {
      "readOnly": null,
      "containerPath": "/ipc",
      "sourceVolume": "node-ipc"
    },
    {
      "readOnly": null,
      "containerPath": "/nix/store",
      "sourceVolume": "node-config"
    },
    {
      "readOnly": null,
      "containerPath": "/data",
      "sourceVolume": "node-testnet-db"
    }
  ],
  "workingDirectory": null,
  "secrets": null,
  "dockerSecurityOptions": null,
  "memory": null,
  "memoryReservation": null,
  "volumesFrom": [],
  "stopTimeout": null,
  "image": "inputoutput/cardano-node:1.18.0",
  "startTimeout": null,
  "firelensConfiguration": null,
  "dependsOn": null,
  "disableNetworking": null,
  "interactive": null,
  "healthCheck": null,
  "essential": true,
  "links": null,
  "hostname": null,
  "extraHosts": null,
  "pseudoTerminal": null,
  "user": null,
  "readonlyRootFilesystem": false,
  "dockerLabels": null,
  "systemControls": null,
  "privileged": null,
  "name": "node"
},
{
  "dnsSearchDomains": null,
  "environmentFiles": null,
  "logConfiguration": {
    "logDriver": "awslogs",
    "secretOptions": null,
    "options": {
      "awslogs-group": "/ecs/wallet-node",
      "awslogs-region": "us-east-1",
      "awslogs-stream-prefix": "ecs"
    }
  },
  "entryPoint": null,
  "portMappings": [
    {
      "hostPort": 8090,
      "protocol": "tcp",
      "containerPort": 8090
    }
  ],
  "command": [
    "serve",
    "--node-socket",
    "/ipc/node.socket",
    "--database",
    "/wallet-db",
    "--listen-address",
    "0.0.0.0",
    "--testnet",
    "/config/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json"
  ],
  "linuxParameters": null,
  "cpu": 0,
  "environment": [
    {
      "name": "NETWORK",
      "value": "testnet"
    }
  ],
  "resourceRequirements": null,
  "ulimits": null,
  "dnsServers": null,
  "mountPoints": [
    {
      "readOnly": null,
      "containerPath": "/ipc",
      "sourceVolume": "node-ipc"
    },
    {
      "readOnly": null,
      "containerPath": "/config",
      "sourceVolume": "node-config"
    },
    {
      "readOnly": null,
      "containerPath": "/wallet-db",
      "sourceVolume": "wallet-testnet-db"
    }
  ],
  "workingDirectory": null,
  "secrets": null,
  "dockerSecurityOptions": null,
  "memory": null,
  "memoryReservation": null,
  "volumesFrom": [],
  "stopTimeout": null,
  "image": "inputoutput/cardano-wallet:latest",
  "startTimeout": null,
  "firelensConfiguration": null,
  "dependsOn": [
    {
      "containerName": "node",
      "condition": "START"
    }
  ],
  "disableNetworking": null,
  "interactive": null,
  "healthCheck": null,
  "essential": true,
  "links": null,
  "hostname": null,
  "extraHosts": null,
  "pseudoTerminal": null,
  "user": null,
  "readonlyRootFilesystem": null,
  "dockerLabels": null,
  "systemControls": null,
  "privileged": null,
  "name": "wallet"
}
],
"placementConstraints": [],
"memory": "4096",
"taskRoleArn": "arn:aws:I am::<AWS ID>:role/ecsTaskRole",
"compatibilities": [
"EC2",
"FARGATE"
],
"taskDefinitionArn": "arn:aws:ecs:us-east-1:<AWS ID>:task-definition/wallet-node-testnet-with-efs",
"family": "wallet-node-testnet-with-efs",
"requiresAttributes": [
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "ecs.capability.execution-role-awslogs"
},
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "ecs.capability.efsAuth"
},
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "ecs.capability.efs"
},
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "com.amazonaws.ecs.capability.task-iam-role"
},
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "ecs.capability.container-ordering"
},
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
},
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
  "targetId": null,
  "targetType": null,
  "value": null,
  "name": "ecs.capability.task-eni"
}
],
"pidMode": null,
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc",
"cpu": "2048",
"revision": 12,
"status": "ACTIVE",
"inferenceAccelerators": null,
"proxyConfiguration": null,
"volumes": [
{
  "efsVolumeConfiguration": {
    "transitEncryptionPort": null,
    "fileSystemId": "<FS ID>",
    "authorizationConfig": {
      "iam": "DISABLED",
      "accessPointId": null
    },
    "transitEncryption": "DISABLED",
    "rootDirectory": "/"
  },
  "name": "node-ipc",
  "host": null,
  "dockerVolumeConfiguration": null
},
{
  "efsVolumeConfiguration": {
    "transitEncryptionPort": null,
    "fileSystemId": "<FS ID>",
    "authorizationConfig": {
      "iam": "DISABLED",
      "accessPointId": null
    },
    "transitEncryption": "DISABLED",
    "rootDirectory": "/"
  },
  "name": "node-config",
  "host": null,
  "dockerVolumeConfiguration": null
},
{
  "efsVolumeConfiguration": {
    "transitEncryptionPort": null,
    "fileSystemId": "<FS ID>",
    "authorizationConfig": {
      "iam": "DISABLED",
      "accessPointId": null
    },
    "transitEncryption": "DISABLED",
    "rootDirectory": "/"
  },
  "name": "node-testnet-db",
  "host": null,
  "dockerVolumeConfiguration": null
},
{
  "efsVolumeConfiguration": {
    "transitEncryptionPort": null,
    "fileSystemId": "<FS ID>",
    "authorizationConfig": {
      "iam": "DISABLED",
      "accessPointId": null
    },
    "transitEncryption": "DISABLED",
    "rootDirectory": "/"
  },
  "name": "wallet-testnet-db",
  "host": null,
  "dockerVolumeConfiguration": null
}
]
}

  3. Create the AWS ECS service with the VPC created, cardano-wallet security group , the tax definition above. Fargate 1.4.0 version.

Cappuccinuo commented 4 years ago

Hey @Kidist-Abraham ,

I can reproduce the issue. Do you have specific reason to mount fs to /nix/store? 

{
      "readOnly": null,
      "containerPath": "/nix/store",
      "sourceVolume": "node-config"
}

Since when I test the cardano-node docker, I found that the config file is under that path, if you mount fs to that path, the docker cannot find the config file.

[ec2-user@ip-172-31-43-114 ~]$ docker run -v /data -e NETWORK=testnet inputoutput/cardano-node:1.18.0
Starting cardano-node run: /nix/store/442brx5kpqdx9z9ccdd5z3zi5km56i05-cardano-node-exe-cardano-node-1.18.0/bin/cardano-node run
--config /nix/store/dfidzx8ggnj6sgzkz8ay7n225sy9nin3-config-0.json
--database-path /data/db
--topology /nix/store/dpajyi2vaychwps1x7d20c2ddls4kf62-topology.yaml
--host-addr 127.0.0.1
--port 3001
--socket-path /ipc/node.socket

..or, once again, in a single line:
/nix/store/442brx5kpqdx9z9ccdd5z3zi5km56i05-cardano-node-exe-cardano-node-1.18.0/bin/cardano-node run --config /nix/store/dfidzx8ggnj6sgzkz8ay7n225sy9nin3-config-0.json --database-path /data/db --topology /nix/store/dpajyi2vaychwps1x7d20c2ddls4kf62-topology.yaml --host-addr 127.0.0.1 --port 3001 --socket-path /ipc/node.socket
Listening on http://127.0.0.1:12798
[a0356538:cardano.node.release:Notice:5] [2020-10-09 07:35:00.37 UTC] CardanoProtocol
[a0356538:cardano.node.networkMagic:Notice:5] [2020-10-09 07:35:00.37 UTC] NetworkMagic 1097911063
[a0356538:cardano.node.version:Notice:5] [2020-10-09 07:35:00.37 UTC] 1.18.0
[a0356538:cardano.node.commit:Notice:5] [2020-10-09 07:35:00.37 UTC] e8fb486e44aceb4b9962d7b3b76849becd510820
[a0356538:cardano.node.ChainDB:Info:5] [2020-10-09 07:35:00.38 UTC] Opened imm db with immutable tip at Origin and epoch 0
[a0356538:cardano.node.ChainDB:Info:5] [2020-10-09 07:35:00.38 UTC] Opened vol db
[a0356538:cardano.node.ChainDB:Info:5] [2020-10-09 07:35:00.38 UTC] Replaying ledger from genesis
[a0356538:cardano.node.ChainDB:Info:5] [2020-10-09 07:35:00.38 UTC] Opened lgr db

We can see that the docker default config file is --config /nix/store/dfidzx8ggnj6sgzkz8ay7n225sy9nin3-config-0.json.

I changed the task definition part to 

"containerDefinitions": [ 
    {
      ...
     // Remove the Command to create the /nix/store
      "mountPoints": [
        ...
        {
          "readOnly": null,
          "containerPath": "/test/store",
          "sourceVolume": "node-config"
        },
        ...
      ],
      ...
      "name": "node"
    },

And now the node can be launched, while the wallet cannot because

Failed to parse genesis configuration. You may want to check the filepath given via --genesis and make sure it points to a valid JSON genesis file. Here's (perhaps) some helpful hint: GenesisDataIOError /config/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json: openBinaryFile: does not exist (No such file or directory)

I think this is due to my fs does not have "/config/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json".

Kidist-Abraham commented 4 years ago

@Cappuccinuo I just tried this and the node is running 👍 But I was mounting to /nix/store so that the wallet can find the path *testnet-byron-genesis.json in /nix/store/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json. You are getting this error because /config in the wallet should have been pointing to /nix/store in the node. I just tried to mount node-config to / and change the command in wallet from/config/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json to /config/nix/store/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json but I am having an error container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:70: creating device nodes caused \\\"errno 524\\\"\""

Kidist-Abraham commented 4 years ago

if you mount fs to that path, the docker cannot find the config file.

Why? The container is mounting /nix/store to node-config, but it should still be able to access /nix/store

Cappuccinuo commented 4 years ago

In the definition,

if you mount fs to that path, the docker cannot find the config file.

Why? The container is mounting /nix/store to node-config, but it should still be able to access /nix/store

In my understanding for the container definition,

sourceVolume (node-config) is the name of the volume to mount, in your case, it is the efsVolume.

containerPath is the path on the container to mount the volume at, in your case, it is the /nix/store. If you mount the fs to the path /nix/store, the folder content will be override by your fs content, that is why I say the docker cannot find the config file.

Kidist-Abraham commented 4 years ago

Shouldn't the fs content be overridden by the content in the container instead? Curios on why this docker-compose file runs fine https://github.com/input-output-hk/cardano-wallet/blob/master/docker-compose.yml Or why this works docker run -v node-config:/nix/store -e NETWORK=testnet inputoutput/cardano-node:1.18.0

anshul0915zinnia commented 3 years ago

we are also facing same issue we are mounting efs to all our task def and use efs as central logging directory so we want every application log there logs in separate application . like we are trying to mount like this

container path efs path
/var/log/abc-app/ /application-logs/abc-app/logs/
/var/log/xyz-app/ /application-logs/xyz-app/logs/

problem is here that we need to create the subdirectories in efs manually

iekdosha commented 9 months ago

we are also facing same issue we are mounting efs to all our task def and use efs as central logging directory so we want every application log there logs in separate application . like we are trying to mount like this

container path efs path /var/log/abc-app/ /application-logs/abc-app/logs/ /var/log/xyz-app/ /application-logs/xyz-app/logs/ problem is here that we need to create the subdirectories in efs manually

I have the same issue as you do