search

crate / docker-crate

Source repository for the official CrateDB Docker image
https://registry.hub.docker.com/_/crate/
Apache License 2.0
49 stars 25 forks source link

How to use this image with local volume mounts (Marathon DCOS) #101

Closed siddharth-ub closed 5 years ago

siddharth-ub commented 6 years ago

Hi, I am trying to use this image in Marathon along with local persistence volumes on DCOS. Since the data folder for crate is chown-ed to crate user , i am unable to read/write to the persistent storage volume once the container is up. The error log :

[o.e.b.ElasticsearchUncaughtExceptionHandler] [SYN-Crate-master1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
    at org.elasticsearch.bootstrap.StartupExceptionProxy.<init>(StartupExceptionProxy.java:31) ~[crate-app-2.2.4.jar:2.2.4]
    at io.crate.bootstrap.CrateDB.init(CrateDB.java:142) ~[crate-app-2.2.4.jar:2.2.4]
    at io.crate.bootstrap.CrateDB.execute(CrateDB.java:118) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:73) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.cli.Command.main(Command.java:88) ~[crate-app-2.2.4.jar:2.2.4]
    at io.crate.bootstrap.CrateDB.main(CrateDB.java:87) ~[crate-app-2.2.4.jar:2.2.4]
    at io.crate.bootstrap.CrateDB.main(CrateDB.java:80) ~[crate-app-2.2.4.jar:2.2.4]
Caused by: java.lang.IllegalStateException: Failed to create node environment
    at org.elasticsearch.node.Node.<init>(Node.java:270) ~[crate-app-2.2.4.jar:2.2.4]
    at io.crate.node.CrateNode.<init>(CrateNode.java:62) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:199) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:199) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:282) ~[crate-app-2.2.4.jar:2.2.4]
    at io.crate.bootstrap.CrateDB.init(CrateDB.java:138) ~[crate-app-2.2.4.jar:2.2.4]
    ... 6 more
Caused by: java.nio.file.AccessDeniedException: /data/nodes
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_151]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_151]
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_151]
    at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_151]
    at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_151]
    at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_151]
    at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_151]
    at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:223) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.node.Node.<init>(Node.java:267) ~[crate-app-2.2.4.jar:2.2.4]
    at io.crate.node.CrateNode.<init>(CrateNode.java:62) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:199) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:199) ~[crate-app-2.2.4.jar:2.2.4]
    at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:282) ~[crate-app-2.2.4.jar:2.2.4]
    at io.crate.bootstrap.CrateDB.init(CrateDB.java:138) ~[crate-app-2.2.4.jar:2.2.4]
    ... 6 more

https://github.com/crate/docker-crate/blob/cc32be338458ce5828e442b1f5cca269bd0ee774/docker-entrypoint.sh#L10

chaudum commented 6 years ago

Hi @siddharth-ub Could you provide your marathon.json for the CrateDB container? CrateDB is run as user crate inside the Docker container, that is why the /data path (inside container) needs to be owned by crate. When you mount a volume of the host to the Docker container, the Docker container will change the permissions of the host file directory that is mounted. Since uid of users inside and outside of the Docker container are different, you probably won't be able read/write on that directory with a crate user on the host system.

siddharth-ub commented 6 years ago

Hi @chaudum ,

Please find the json below : 

{
  "id": "/cratem1",
  "cmd": null,
  "cpus": 2,
  "mem": 4096,
  "disk": 10240,
  "instances": 0,
  "acceptedResourceRoles": [
    "*"
  ],
  "container": {
    "type": "DOCKER",
    "volumes": [
      {
        "containerPath": "/crate/data",
        "hostPath": "data",
        "mode": "RW"
      },
      {
        "containerPath": "data",
        "mode": "RW",
        "persistent": {
          "size": 10000,
          "type": "root",
          "constraints": []
        }
      }
    ],
    "docker": {
      "image": "registry.symphoni.io/crate/cratem1:v84",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 4200,
          "hostPort": 0,
          "servicePort": 10288,
          "protocol": "tcp",
          "labels": {}
        },
        {
          "containerPort": 4301,
          "hostPort": 0,
          "servicePort": 10285,
          "protocol": "tcp",
          "labels": {}
        },
        {
          "containerPort": 5432,
          "hostPort": 0,
          "servicePort": 10289,
          "protocol": "tcp",
          "labels": {}
        }
      ],
      "privileged": false,
      "parameters": [
        {
          "key": "user",
          "value": "crate"
        }
      ],
      "forcePullImage": false
    }
  },
  "env": {
    "bo.soa.config.enable-message-builder": "false",
    "bo.soa.config.orm-endpoint": "orm",
    "bo.custom.security-crypto-url": "https://securitycrypto.invent.symphoni.io/KeyExchange/SecretKey",
    "spring.datasource.url": "jdbc:com.nuodb://x.x.x.x:48004/pass_Application?schema=BOSUITE",
    "bo.soa.config.env": "PRODUCTION",
    "security.oauth2.resource.user-info-uri": "https://identityserver.invent.symphoni.io/",
    "bo.custom.environment": "PRODUCTION",
    "bo.custom.database-url": "jdbc:com.nuodb://x.x.x.x:48004/pass_Application?schema=BOSUITE",
    "bo.soa.config.enable-middle-tier": "false",
    "bo.soa.config.service-url": "https://serviceurl.invent.symphoni.io/serviceURL/",
    "bo.soa.security.secret-key-url": "https://securitycrypto.invent.symphoni.io/KeyExchange/SecretKey",
    "bo.custom.orm-name": "orm"
  },
  "healthChecks": [
    {
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3,
      "delaySeconds": 15,
      "command": {
        "value": "bash -c \\\"</dev/tcp/$HOST/$PORT0\\\""
      },
      "protocol": "COMMAND"
    }
  ],
  "labels": {
    "HAPROXY_0_MODE": "http",
    "HAPROXY_GROUP": "external",
    "HAPROXY_1_PORT": "4301",
    "HAPROXY_2_PORT": "5432",
    "HAPROXY_1_MODE": "tcp",
    "HAPROXY_2_MODE": "tcp",
    "HAPROXY_0_VHOST": "cratem1.invent.symphoni.io"
  },
  "portDefinitions": [
    {
      "port": 10288,
      "protocol": "tcp",
      "name": "default",
      "labels": {}
    },
    {
      "port": 10285,
      "protocol": "tcp",
      "labels": {}
    },
    {
      "port": 10289,
      "protocol": "tcp",
      "labels": {}
    }
  ],
  "uris": [
    "http://dockerconfig.invent.symphoni.io/public/docker.tar.gz"
  ],
  "fetch": [
    {
      "uri": "http://dockerconfig.invent.symphoni.io/public/docker.tar.gz",
      "extract": true,
      "executable": false,
      "cache": false
    }
  ],
  "upgradeStrategy": {
    "minimumHealthCapacity": 0.5,
    "maximumOverCapacity": 0
  }
}
chaudum commented 6 years ago

I'm not too familiar with Mesos, but from reading the documentation, a MOUNT disk would be more suitable for CrateDB persistent storage: http://mesos.apache.org/documentation/latest/multiple-disk/#mount-disks

A Mount disk is an auxiliary disk resource provided by the operator. This cannot be carved up into smaller chunks by frameworks. This lack of flexibility allows operators to provide assurances to frameworks that they will have exclusive access to the disk device. Common uses for this kind of disk include database storage, write-ahead logs, or other performance-critical applications.

Make sure the mount path is read/writable by the Mesos role that is running CrateDB.

mfussenegger commented 5 years ago

Looks like the question has been answered and there hasn't been any feedback to it. So I'm closing this.

Please re-open if this is still an issue.