search

Sage-Bionetworks / SynapseWorkflowOrchestrator

Apache License 2.0
3 stars 2 forks source link

Can't specify the specific tag for the docker image of WORKFLOW_ENGINE_DOCKER_IMAGE #1

Open thomasyu888 opened 5 years ago

thomasyu888 commented 5 years ago 

compose.config.config.find: Using configuration files: ./docker-compose.yaml
WARNING: compose.config.environment.__getitem__: The NOTIFICATION_PRINCIPAL_ID variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The SHARE_RESULTS_IMMEDIATELY variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The DATA_UNLOCK_SYNAPSE_PRINCIPAL_ID variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The MAX_CONCURRENT_WORKFLOWS variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The SUBMITTER_NOTIFICATION_MASK variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The RUN_WORKFLOW_CONTAINER_IN_PRIVILEGED_MODE variable is not set. Defaulting to a blank string.
docker.utils.config.find_config_file: Trying paths: ['/home/tyu/.docker/config.json', '/home/tyu/.dockercfg']
docker.utils.config.find_config_file: No config file found
docker.utils.config.find_config_file: Trying paths: ['/home/tyu/.docker/config.json', '/home/tyu/.dockercfg']
docker.utils.config.find_config_file: No config file found
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/version HTTP/1.1" 200 589
compose.cli.command.get_client: docker-compose version 1.24.1, build 4667896b
docker-py version: 3.7.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018
compose.cli.command.get_client: Docker base_url: http+docker://localhost
compose.cli.command.get_client: Docker version: Platform={'Name': ''}, Components=[{'Name': 'Engine', 'Version': '18.06.1-ce', 'Details': {'ApiVersion': '1.38', 'Arch': 'amd64', 'BuildTime': '2019-07-01T18:53:20.000000000+00:00', 'Experimental': 'false', 'GitCommit': 'e68fc7a/18.06.1-ce', 'GoVersion': 'go1.10.3', 'KernelVersion': '4.14.123-111.109.amzn2.x86_64', 'MinAPIVersion': '1.12', 'Os': 'linux'}}], Version=18.06.1-ce, ApiVersion=1.38, MinAPIVersion=1.12, GitCommit=e68fc7a/18.06.1-ce, GoVersion=go1.10.3, Os=linux, Arch=amd64, KernelVersion=4.14.123-111.109.amzn2.x86_64, BuildTime=2019-07-01T18:53:20.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('workfloworchestrator_default')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/networks/workfloworchestrator_default HTTP/1.1" 404 61
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('workfloworchestrator_shared')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/volumes/workfloworchestrator_shared HTTP/1.1" 404 62
compose.cli.verbose_proxy.proxy_callable: docker info <- ()
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/info HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64',
 'BridgeNfIp6tables': True,
 'BridgeNfIptables': True,
 'CPUSet': True,
 'CPUShares': True,
 'CgroupDriver': 'cgroupfs',
 'ClusterAdvertise': '',
 'ClusterStore': '',
 'ContainerdCommit': {'Expected': '468a545b9edcd5932818eb9de8e72413e616e86e',
                      'ID': '468a545b9edcd5932818eb9de8e72413e616e86e'},
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('workflow_orchestrator_default')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/networks/workflow_orchestrator_default HTTP/1.1" 200 581
compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {'Attachable': True,
 'ConfigFrom': {'Network': ''},
 'ConfigOnly': False,
 'Containers': {},
 'Created': '2019-10-10T19:38:05.587183038Z',
 'Driver': 'bridge',
 'EnableIPv6': False,
 'IPAM': {'Config': [{'Gateway': '172.19.0.1', 'Subnet': '172.19.0.0/16'}],
          'Driver': 'default',
          'Options': None},
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('workflow_orchestrator_shared')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/volumes/workflow_orchestrator_shared HTTP/1.1" 200 336
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume -> {'CreatedAt': '2019-10-10T19:49:11Z',
 'Driver': 'local',
 'Labels': {'com.docker.compose.project': 'workflow_orchestrator',
            'com.docker.compose.version': '1.24.1',
            'com.docker.compose.volume': 'shared'},
 'Mountpoint': '/var/lib/docker/volumes/workflow_orchestrator_shared/_data',
 'Name': 'workflow_orchestrator_shared',
 'Options': None,
 'Scope': 'local'}
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('workflow_orchestrator_shared')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/volumes/workflow_orchestrator_shared HTTP/1.1" 200 336
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume -> {'CreatedAt': '2019-10-10T19:49:11Z',
 'Driver': 'local',
 'Labels': {'com.docker.compose.project': 'workflow_orchestrator',
            'com.docker.compose.version': '1.24.1',
            'com.docker.compose.volume': 'shared'},
 'Mountpoint': '/var/lib/docker/volumes/workflow_orchestrator_shared/_data',
 'Name': 'workflow_orchestrator_shared',
 'Options': None,
 'Scope': 'local'}
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=workflow_orchestrator', 'com.docker.compose.oneoff=False']})
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dworkflow_orchestrator%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 3
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=workfloworchestrator', 'com.docker.compose.oneoff=False']})
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dworkfloworchestrator%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 3
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=workflow_orchestrator', 'com.docker.compose.service=workflow-orchestrator', 'com.docker.compose.oneoff=False']})
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/containers/json?limit=-1&all=1&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dworkflow_orchestrator%22%2C+%22com.docker.compose.service%3Dworkflow-orchestrator%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 1591
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('a1f5907c5bac48fbcb2abaaeeead7154a44da7f99084dc6c610b41bed331b082')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/containers/a1f5907c5bac48fbcb2abaaeeead7154a44da7f99084dc6c610b41bed331b082/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['/bin/sh',
          '-c',
          'exec mvn exec:java '
          '-DentryPoint=org.sagebionetworks.WorkflowOrchestrator'],
 'Config': {'ArgsEscaped': True,
            'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['/bin/sh',
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('sagebionetworks/synapse-workflow-orchestrator:1.0')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/images/sagebionetworks/synapse-workflow-orchestrator:1.0/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64',
 'Author': '',
 'Comment': '',
 'Config': {'ArgsEscaped': True,
            'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['/bin/sh',
                    '-c',
                    'exec mvn exec:java '
...
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=workflow_orchestrator', 'com.docker.compose.service=workflow-orchestrator', 'com.docker.compose.oneoff=False']})
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/containers/json?limit=-1&all=1&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dworkflow_orchestrator%22%2C+%22com.docker.compose.service%3Dworkflow-orchestrator%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 1591
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('sagebionetworks/synapse-workflow-orchestrator:1.0')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/images/sagebionetworks/synapse-workflow-orchestrator:1.0/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64',
 'Author': '',
 'Comment': '',
 'Config': {'ArgsEscaped': True,
            'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['/bin/sh',
                    '-c',
                    'exec mvn exec:java '
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('a1f5907c5bac48fbcb2abaaeeead7154a44da7f99084dc6c610b41bed331b082')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/containers/a1f5907c5bac48fbcb2abaaeeead7154a44da7f99084dc6c610b41bed331b082/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['/bin/sh',
          '-c',
          'exec mvn exec:java '
          '-DentryPoint=org.sagebionetworks.WorkflowOrchestrator'],
 'Config': {'ArgsEscaped': True,
            'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['/bin/sh',
...
compose.parallel.feed_queue: Pending: {<Service: workflow-orchestrator>}
compose.parallel.feed_queue: Starting producer thread for <Service: workflow-orchestrator>
Starting workflow_orchestrator_workflow-orchestrator_1 ... 
compose.parallel.feed_queue: Pending: {<Container: workflow_orchestrator_workflow-orchestrator_1 (a1f590)>}
compose.parallel.feed_queue: Starting producer thread for <Container: workflow_orchestrator_workflow-orchestrator_1 (a1f590)>
compose.cli.verbose_proxy.proxy_callable: docker attach <- ('a1f5907c5bac48fbcb2abaaeeead7154a44da7f99084dc6c610b41bed331b082', stdout=True, stderr=True, stream=True)
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.25/containers/a1f5907c5bac48fbcb2abaaeeead7154a44da7f99084dc6c610b41bed331b082/attach?logs=0&stdout=1&stderr=1&stream=1 HTTP/1.1" 101 0
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/containers/a1f5907c5bac48fbcb2abaaeeead7154a44da7f99084dc6c610b41bed331b082/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker attach -> <docker.types.daemon.CancellableStream object at 0x7f651f6d15c0>
compose.cli.verbose_proxy.proxy_callable: docker start <- ('a1f5907c5bac48fbcb2abaaeeead7154a44da7f99084dc6c610b41bed331b082')
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.25/containers/a1f5907c5bac48fbcb2abaaeeead7154a44da7f99084dc6c610b41bed331b082/start HTTP/1.1" 204 0
compose.cli.verbose_proxy.proxy_callable: docker start -> None
compose.parallel.parallel_execute_iter: Finished processing: <Container: workfloStarting workflow_orchestrator_workflow-orchestrator_1 ... done
compose.parallel.feed_queue: Pending: set()
compose.parallel.parallel_execute_iter: Finished processing: <Service: workflow-orchestrator>
compose.parallel.feed_queue: Pending: set()
Attaching to workflow_orchestrator_workflow-orchestrator_1
compose.cli.verbose_proxy.proxy_callable: docker events <- (filters={'label': ['com.docker.compose.project=workflow_orchestrator', 'com.docker.compose.oneoff=False']}, decode=True)
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.25/events?filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dworkflow_orchestrator%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker events -> <docker.types.daemon.CancellableStream object at 0x7f651f6dc7f0>
workflow-orchestrator_1  | [INFO] Scanning for projects...
workflow-orchestrator_1  | [INFO] 
workflow-orchestrator_1  | [INFO] --------------< org.sagebionetworks:WorkflowOrchestrator >--------------
workflow-orchestrator_1  | [INFO] Building WorkflowOrchestrator 1.0-SNAPSHOT
workflow-orchestrator_1  | [INFO] --------------------------------[ jar ]---------------------------------
workflow-orchestrator_1  | [INFO] 
workflow-orchestrator_1  | [INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) > validate @ WorkflowOrchestrator >>>
workflow-orchestrator_1  | [INFO] 
workflow-orchestrator_1  | [INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) < validate @ WorkflowOrchestrator <<<
workflow-orchestrator_1  | [INFO] 
workflow-orchestrator_1  | [INFO] 
workflow-orchestrator_1  | [INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ WorkflowOrchestrator ---
workflow-orchestrator_1  | ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
workflow-orchestrator_1  | [org.sagebionetworks.WorkflowOrchestrator.main()] INFO org.sagebionetworks.WorkflowOrchestrator - Precheck completed successfully.
workflow-orchestrator_1  | [org.sagebionetworks.WorkflowOrchestrator.main()] INFO org.sagebionetworks.WorkflowOrchestrator - Top level loop: checking progress or starting new job.
workflow-orchestrator_1  | 19:50:44.894 [org.sagebionetworks.WorkflowOrchestrator.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-orchestrator_1  | Oct 10, 2019 7:50:44 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-orchestrator_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230 already exists with a parentId: syn4990358
workflow-orchestrator_1  | Oct 10, 2019 7:50:46 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-orchestrator_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9693821 already exists with a parentId: syn16966592
workflow-orchestrator_1  | 19:50:46.057 [org.sagebionetworks.WorkflowOrchestrator.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-orchestrator_1  | 19:50:46.286 [org.sagebionetworks.WorkflowOrchestrator.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-orchestrator_1  | Oct 10, 2019 7:50:46 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-orchestrator_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 3324230_LOCKED already exists with a parentId: syn4990358
workflow-orchestrator_1  | 19:50:46.515 [org.sagebionetworks.WorkflowOrchestrator.main()] ERROR org.sagebionetworks.client.SynapseProfileProxy - java.lang.reflect.InvocationTargetException
workflow-orchestrator_1  | Oct 10, 2019 7:50:46 PM org.sagebionetworks.ExponentialBackoffRunner execute
workflow-orchestrator_1  | SEVERE: Found status code 409. Will not retry: Status Code: 409 message: An entity with the name: 9693821 already exists with a parentId: syn16966595
workflow-orchestrator_1  | [org.sagebionetworks.WorkflowOrchestrator.main()] INFO org.sagebionetworks.WorkflowManagerDocker - Mounting: /var/run/docker.sock
workflow-orchestrator_1  | [org.sagebionetworks.WorkflowOrchestrator.main()] INFO org.sagebionetworks.WorkflowManagerDocker - workingDir: /var/lib/docker/volumes/workflow_orchestrator_shared/_data/15a49baa-5840-4b3f-b512-41e47150d672
workflow-orchestrator_1  | [org.sagebionetworks.WorkflowOrchestrator.main()] INFO org.sagebionetworks.WorkflowManagerDocker - toil cmd: [toil-cwl-runner, --defaultMemory, 100M, --retryCount, 0, --defaultDisk, 1000000, --workDir, /var/lib/docker/volumes/workflow_orchestrator_shared/_data/15a49baa-5840-4b3f-b512-41e47150d672, --noLinkImports, /var/lib/docker/volumes/workflow_orchestrator_shared/_data/15a49baa-5840-4b3f-b512-41e47150d672/ChallengeWorkflowTemplates-develop/scoring_harness_workflow.cwl, /var/lib/docker/volumes/workflow_orchestrator_shared/_data/15a49baa-5840-4b3f-b512-41e47150d672/TMP12502553979814642737.yaml]
workflow-orchestrator_1  | [dockerjava-jaxrs-async-0] ERROR com.github.dockerjava.core.async.ResultCallbackTemplate - Error during callback
workflow-orchestrator_1  | com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://registry-1.docker.io/v2/sagebionetworks/synapse-workflow-orchestrator-toil/manifests/1.0: unauthorized: incorrect username or password"}
workflow-orchestrator_1  | 
workflow-orchestrator_1  |  at com.github.dockerjava.jaxrs.filter.ResponseStatusExceptionFilter.filter(ResponseStatusExceptionFilter.java:53)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:140)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:128)
workflow-orchestrator_1  |  at org.glassfish.jersey.process.internal.Stages.process(Stages.java:171)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:257)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
workflow-orchestrator_1  |  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
workflow-orchestrator_1  |  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
workflow-orchestrator_1  |  at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
workflow-orchestrator_1  |  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
workflow-orchestrator_1  |  at com.github.dockerjava.jaxrs.async.POSTCallbackNotifier.response(POSTCallbackNotifier.java:29)
workflow-orchestrator_1  |  at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:50)
workflow-orchestrator_1  |  at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:24)
workflow-orchestrator_1  |  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
workflow-orchestrator_1  |  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
workflow-orchestrator_1  |  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
workflow-orchestrator_1  |  at java.base/java.lang.Thread.run(Thread.java:834)
workflow-orchestrator_1  | [dockerjava-jaxrs-async-1] ERROR com.github.dockerjava.core.async.ResultCallbackTemplate - Error during callback
workflow-orchestrator_1  | com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://registry-1.docker.io/v2/sagebionetworks/synapse-workflow-orchestrator-toil/manifests/1.0: unauthorized: incorrect username or password"}
workflow-orchestrator_1  | 
workflow-orchestrator_1  |  at com.github.dockerjava.jaxrs.filter.ResponseStatusExceptionFilter.filter(ResponseStatusExceptionFilter.java:53)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:140)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:128)
workflow-orchestrator_1  |  at org.glassfish.jersey.process.internal.Stages.process(Stages.java:171)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:257)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701)
workflow-orchestrator_1  |  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
workflow-orchestrator_1  |  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
workflow-orchestrator_1  |  at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
workflow-orchestrator_1  |  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448)
workflow-orchestrator_1  |  at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:349)
workflow-orchestrator_1  |  at com.github.dockerjava.jaxrs.async.POSTCallbackNotifier.response(POSTCallbackNotifier.java:29)
workflow-orchestrator_1  |  at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:50)
workflow-orchestrator_1  |  at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:24)
workflow-orchestrator_1  |  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
workflow-orchestrator_1  |  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
workflow-orchestrator_1  |  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
workflow-orchestrator_1  |  at java.base/java.lang.Thread.run(Thread.java:834)
thomasyu888 commented 5 years ago

Specifically there is an error that says: workflow-orchestrator_1 | com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://registry-1.docker.io/v2/sagebionetworks/synapse-workflow-orchestrator-toil/manifests/1.0: unauthorized: incorrect username or password"}

brucehoff commented 5 years ago

@thomasyu888 can you do one of the following?

1) before running the Orchestrator, run

docker pull sagebionetworks/synapse-workflow-orchestrator-toil:1.0

or

2) add to your .env

DOCKERHUB_USERNAME=<your DockerHub name>
DOCKERHUB_PASSWORD=<your DockerHub password>

and add to docker-compose.yaml

environment:
   ....
   - DOCKERHUB_USERNAME=${DOCKERHUB_USERNAME}
   - DOCKERHUB_PASSWORD=${DOCKERHUB_PASSWORD}
thomasyu888 commented 5 years ago

The interesting part about this is that I don't need to log into dockerhub to pull public images. So there is something specific about Get https://registry-1.docker.io/v2/sagebionetworks/synapse-workflow-orchestrator-toil/manifests/1.0 that requires me to be logged into dockerhub?

Edit. Tried 1 - same error Tried 2 - same error 1 + 2 - same error

brucehoff commented 5 years ago

I should not have suggested (1), it won't help. That is, the Orchestrator will try to pull the image regardless of whether you already have it.

I checked the Java Docker client used by the Orchestrator and verified it will pull a public repo' when no user name or password is given. I tested several repo's including sagebionetworks/synapse-workflow-orchestrator-toil:1.0. Also I checked that sagebionetworks/synapse-workflow-orchestrator-toil:1.0 is public. I also verified that the Orchestrator will fail to download a private image when I don't pass cred's, so it's not that it's getting cred's elsewhere.

I further verified that if I pass the wrong credentials when pulling a public repository it still works, so the Registry must know not to check credentials when pulling a public repo'.

thomasyu888 commented 5 years ago

Hm. Here is my workflow:

  1. docker --version
Docker version 18.06.1-ce, build e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
  2. docker logout
Removing login credentials for https://index.docker.io/v1/
  3. docker images
REPOSITORY                                           TAG                 IMAGE ID            CREATED             SIZE
sagebionetworks/synapse-workflow-orchestrator        1.0                 cc4f371498bc        4 hours ago         966MB
sagebionetworks/synapse-workflow-orchestrator-toil   1.0                 85858597c42a        4 hours ago         2.01GB
sagebionetworks/synapse-workflow-orchestrator-wes    1.0                 005f779dcdc7        5 hours ago         1.25GB
ubuntu                                               latest              2ca708c1c9cc        3 weeks ago         64.2MB
  4. docker rm $(docker ps -aq)
docker rmi $(docker images -q)
  5. docker-compose up

I still receive the same error. I also removed DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD

Here is my .env


#WES_ENPDOINT=http://localhost:8082
DOCKER_ENGINE_URL=unix:///var/run/docker.sock
#DOCKER_ENGINE_URL=
#WES_SHARED_DIR_PROPERTY=/home/tyu/orchestrator
SYNAPSE_USERNAME=xxxxx
SYNAPSE_PASSWORD=xxxxx
WORKFLOW_OUTPUT_ROOT_ENTITY_ID=syn4990358
EVALUATION_TEMPLATES={"9614111": "syn20946645"}
TOIL_CLI_OPTIONS=--defaultMemory 100M --retryCount 0 --defaultDisk 1000000
COMPOSE_PROJECT_NAME=workflow_orchestrator
#NOTIFICATION_PRINCIPAL_ID=
#SHARE_RESULTS_IMMEDIATELY=
#DATA_UNLOCK_SYNAPSE_PRINCIPAL_ID=
#WORKFLOW_ENGINE_DOCKER_IMAGE=
#MAX_CONCURRENT_WORKFLOWS=10
#RUN_WORKFLOW_CONTAINER_IN_PRIVILEGED_MODE=```
brucehoff commented 5 years ago

I can reproduce the problem, by running the containerized Orchestrator:

workflow-orchestrator_1  | com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://registry-1.docker.io/v2/sagebionetworks/synapse-workflow-orchestrator-toil/manifests/1.0: unauthorized: incorrect username or password"}

I can verify that passing my correct DockerHub credentials does not fix the problem.

When I try to reproduce the error by running the internal Java library it works fine!

brucehoff commented 5 years ago

More insight: The problem goes away if you switch to the 'old' Toil image, i.e. changing from

sagebionetworks/synapse-workflow-orchestrator-toil:1.0

to 

sagebionetworks/synapseworkflowhook-toil
thomasyu888 commented 5 years ago

Maybe it has something to do with defining a tag?

brucehoff commented 5 years ago

I had the same thought. I will test that and also will try upgrading to a newer version of the Java Docker client.

brucehoff commented 5 years ago

Maybe it has something to do with defining a tag?

Yes! I pushed sagebionetworks/synapse-workflow-orchestrator-toil:1.0 to DockerHub as sagebionetworks/synapse-workflow-orchestrator-toil and used the latter with the Orchestrator. It worked, no problem.

brucehoff commented 5 years ago

I upgraded the Java Docker client to the latest version, but nothing changed.

I tried replacing sagebionetworks/synapse-workflow-orchestrator-toil with sagebionetworks/synapse-workflow-orchestrator-toil:latest which is a synonym. While the version with no tag works, the version with the 'latest' tag does not. 

workflow-orchestrator_1  | com.github.dockerjava.api.exception.InternalServerErrorException: Get https://registry-1.docker.io/v2/sagebionetworks/synapse-workflow-orchestrator-toil/manifests/latest: unauthorized: incorrect username or password

What's maddening is that there is no problem when I run the Docker Java client from my IDE, everything's fine. It only breaks when running in a container. That tells me that in the 'IDE case' the library is accessing registry credentials on my machine. I broke my stored registry password, confirmed by trying to 'docker pull' from the command line. But the 'IDE case' still works.