jsbroks / coco-annotator

:pencil2: Web-based image segmentation tool for object detection, localization, and keypoints
MIT License
2.06k stars 451 forks source link

Backend: MaskRCNN Logging issues #235

Open kevdagoat opened 4 years ago

kevdagoat commented 4 years ago

After getting it sucessfully running, in a production environment i would like to see the log from the MaskRCNN backend (to add more functionality).

I see that https://github.com/jsbroks/coco-annotator/pull/201/commits/93e9b2eaf8b0f3f0c63e47cc6f7d74d2ba89c346 tried to fix this but it is still not working for me

docker-compose.yml:

version: "3"
volumes:
  mongodb_data:
    external: false

networks:
  cocoannotator:
    external: false

services:
  webserver:
    image: jsbroks/coco-annotator:webserver-stable
    container_name: annotator_webclient
    restart: always
    ports:
      - "5000:5000"
    environment:
      - SECRET_KEY=RandomSecretKeyHere
      - FILE_WATCHER=true
      - MASK_RCNN_FILE=/models/corrosion_latest.h5
      - MASK_RCNN_CLASSES=BG,corrosion
      - FLASK_ENV=development
      - DEBUG=true
    volumes:
      - "/addons/coco_annotator_files/datasets:/datasets"
      - "/addons/coco_annotator_files/models:/models"
    depends_on:
      - database
      - workers
    networks:
      - cocoannotator
  workers:
    container_name: annotator_workers
    image: jsbroks/coco-annotator:workers-stable
    volumes:
      - "/addons/coco_annotator_files/datasets:/datasets"
    depends_on:
      - messageq
      - database
    networks:
      - cocoannotator
  messageq:
    image: rabbitmq:3
    container_name: annotator_message_q
    environment:
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password
      - DEBUG=true
    networks:
      - cocoannotator
  database:
    image: mongo:4
    container_name: annotator_mongodb
    restart: always
    environment:
      - MONGO_DATA_DIR=/data/db
      - MONGO_LOG_DIR=/dev/null
    volumes:
      - "mongodb_data:/data/db"
    command: "mongod --smallfiles --logpath=/dev/null"
    networks:
      - cocoannotator

Log output:

annotator_workers | fatal: Not a git repository (or any of the parent directories): .git
annotator_webclient | [2019-08-08 09:53:21 +0000] [6] [DEBUG] Current configuration:
annotator_webclient |   config: webserver/gunicorn_config.py
annotator_webclient |   bind: ['0.0.0.0:5000']
annotator_webclient |   backlog: 2048
annotator_webclient |   workers: 1
annotator_webclient |   worker_class: eventlet
annotator_webclient |   threads: 1
annotator_webclient |   worker_connections: 1000
annotator_webclient |   max_requests: 0
annotator_webclient |   max_requests_jitter: 0
annotator_webclient |   timeout: 180
annotator_webclient |   graceful_timeout: 30
annotator_webclient |   keepalive: 2
annotator_webclient |   limit_request_line: 4094
annotator_webclient |   limit_request_fields: 100
annotator_webclient |   limit_request_field_size: 8190
annotator_webclient |   reload: True
annotator_webclient |   reload_engine: auto
annotator_webclient |   reload_extra_files: []
annotator_webclient |   spew: False
annotator_webclient |   check_config: False
annotator_webclient |   preload_app: False
annotator_webclient |   sendfile: False
annotator_webclient |   reuse_port: False
annotator_webclient |   chdir: /workspace
annotator_webclient |   daemon: False
annotator_webclient |   raw_env: []
annotator_webclient |   pidfile: None
annotator_webclient |   worker_tmp_dir: None
annotator_webclient |   user: 0
annotator_webclient |   group: 0
annotator_webclient |   umask: 0
annotator_webclient |   initgroups: False
annotator_webclient |   tmp_upload_dir: None
annotator_webclient |   secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
annotator_webclient |   forwarded_allow_ips: ['127.0.0.1']
annotator_webclient |   accesslog: None
annotator_webclient |   disable_redirect_access_to_syslog: False
annotator_webclient |   access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
annotator_webclient |   errorlog: -
annotator_webclient |   loglevel: debug
annotator_webclient |   capture_output: False
annotator_webclient |   logger_class: gunicorn.glogging.Logger
annotator_webclient |   logconfig: None
annotator_webclient |   logconfig_dict: {}
annotator_webclient |   syslog_addr: udp://localhost:514
annotator_webclient |   syslog: False
annotator_webclient |   syslog_prefix: None
annotator_webclient |   syslog_facility: user
annotator_webclient |   enable_stdio_inheritance: False
annotator_webclient |   statsd_host: None
annotator_webclient |   statsd_prefix:
annotator_webclient |   proc_name: None
annotator_webclient |   default_proc_name: webserver:app
annotator_webclient |   pythonpath: None
annotator_webclient |   paste: None
annotator_webclient |   on_starting: <function OnStarting.on_starting at 0x7f72369df2f0>
annotator_webclient |   on_reload: <function OnReload.on_reload at 0x7f72369df400>
annotator_webclient |   when_ready: <function WhenReady.when_ready at 0x7f72369df510>
annotator_webclient |   pre_fork: <function Prefork.pre_fork at 0x7f72369df620>
annotator_webclient |   post_fork: <function Postfork.post_fork at 0x7f72369df730>
annotator_webclient |   post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f72369df840>
annotator_webclient |   worker_int: <function WorkerInt.worker_int at 0x7f72369df950>
annotator_webclient |   worker_abort: <function WorkerAbort.worker_abort at 0x7f72369dfa60>
annotator_webclient |   pre_exec: <function PreExec.pre_exec at 0x7f72369dfb70>
annotator_webclient |   pre_request: <function PreRequest.pre_request at 0x7f72369dfc80>
annotator_webclient |   post_request: <function PostRequest.post_request at 0x7f72369dfd08>
annotator_webclient |   child_exit: <function ChildExit.child_exit at 0x7f72369dfe18>
annotator_webclient |   worker_exit: <function WorkerExit.worker_exit at 0x7f72369dff28>
annotator_webclient |   nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f723675c0d0>
annotator_webclient |   on_exit: <function OnExit.on_exit at 0x7f723675c1e0>
annotator_webclient |   proxy_protocol: False
annotator_webclient |   proxy_allow_ips: ['127.0.0.1']
annotator_webclient |   keyfile: None
annotator_webclient |   certfile: None
annotator_webclient |   ssl_version: 2
annotator_webclient |   cert_reqs: 0
annotator_webclient |   ca_certs: None
annotator_webclient |   suppress_ragged_eofs: True
annotator_webclient |   do_handshake_on_connect: False
annotator_webclient |   ciphers: TLSv1
annotator_webclient |   raw_paste_global_conf: []
annotator_webclient | [2019-08-08 09:53:21 +0000] [6] [INFO] Starting gunicorn 19.9.0
annotator_webclient | [2019-08-08 09:53:21 +0000] [6] [DEBUG] Arbiter booted
annotator_webclient | [2019-08-08 09:53:21 +0000] [6] [INFO] Listening at: http://0.0.0.0:5000 (6)
annotator_webclient | [2019-08-08 09:53:21 +0000] [6] [INFO] Using worker: eventlet
annotator_webclient | [2019-08-08 09:53:21 +0000] [10] [INFO] Booting worker with pid: 10
annotator_webclient | [2019-08-08 09:53:21 +0000] [6] [DEBUG] 1 workers
annotator_workers | /usr/local/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
annotator_workers | absolutely not recommended!
annotator_workers |
annotator_workers | Please specify a different user using the --uid option.
annotator_workers |
annotator_workers | User information: uid=0 euid=0 gid=0 egid=0
annotator_workers |
annotator_workers |   uid=uid, euid=euid, gid=gid, egid=egid,
annotator_workers | [2019-08-08 09:53:22,202: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: [Errno 111] Connection refused.
annotator_workers | Trying again in 2.00 seconds...
annotator_workers |
annotator_message_q | 2019-08-08 09:53:22.771 [info] <0.8.0> Feature flags: list of feature flags found:
annotator_message_q | 2019-08-08 09:53:22.771 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
annotator_message_q | 2019-08-08 09:53:22.836 [info] <0.266.0>
annotator_message_q |  Starting RabbitMQ 3.7.17 on Erlang 22.0.7
annotator_message_q |  Copyright (C) 2007-2019 Pivotal Software, Inc.
annotator_message_q |  Licensed under the MPL.  See https://www.rabbitmq.com/
annotator_message_q |
annotator_message_q |   ##  ##
annotator_message_q |   ##  ##      RabbitMQ 3.7.17. Copyright (C) 2007-2019 Pivotal Software, Inc.
annotator_message_q |   ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
annotator_message_q |   ######  ##
annotator_message_q |   ##########  Logs: <stdout>
annotator_message_q |
annotator_message_q |               Starting broker...
annotator_message_q | 2019-08-08 09:53:22.837 [info] <0.266.0>
annotator_message_q |  node           : rabbit@edc67d7ed6fc
annotator_message_q |  home dir       : /var/lib/rabbitmq
annotator_message_q |  config file(s) : /etc/rabbitmq/rabbitmq.conf
annotator_message_q |  cookie hash    : dUJut5WwLn6HOxdneFWT5g==
annotator_message_q |  log(s)         : <stdout>
annotator_message_q |  database dir   : /var/lib/rabbitmq/mnesia/rabbit@edc67d7ed6fc
annotator_message_q | 2019-08-08 09:53:22.845 [info] <0.266.0> Running boot step pre_boot defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.845 [info] <0.266.0> Running boot step rabbit_core_metrics defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.846 [info] <0.266.0> Running boot step rabbit_alarm defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.850 [info] <0.293.0> Memory high watermark set to 25779 MiB (27031968153 bytes) of 64449 MiB (67579920384 bytes) total
annotator_message_q | 2019-08-08 09:53:22.854 [info] <0.295.0> Enabling free disk space monitoring
annotator_message_q | 2019-08-08 09:53:22.854 [info] <0.295.0> Disk free limit set to 50MB
annotator_message_q | 2019-08-08 09:53:22.857 [info] <0.266.0> Running boot step code_server_cache defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.857 [info] <0.266.0> Running boot step file_handle_cache defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.857 [info] <0.298.0> Limiting to approx 1048476 file handles (943626 sockets)
annotator_message_q | 2019-08-08 09:53:22.857 [info] <0.299.0> FHC read buffering:  OFF
annotator_message_q | 2019-08-08 09:53:22.857 [info] <0.299.0> FHC write buffering: ON
annotator_message_q | 2019-08-08 09:53:22.858 [info] <0.266.0> Running boot step worker_pool defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.858 [info] <0.268.0> Will use 40 processes for default worker pool
annotator_message_q | 2019-08-08 09:53:22.858 [info] <0.268.0> Starting worker pool 'worker_pool' with 40 processes in it
annotator_message_q | 2019-08-08 09:53:22.860 [info] <0.266.0> Running boot step database defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.861 [info] <0.266.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q | 2019-08-08 09:53:22.879 [info] <0.266.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Running boot step database_sync defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Running boot step feature_flags defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Running boot step codec_correctness_check defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Running boot step external_infrastructure defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Running boot step rabbit_registry defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Running boot step rabbit_queue_location_random defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Running boot step rabbit_event defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.880 [info] <0.266.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.881 [info] <0.266.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.881 [info] <0.266.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.881 [info] <0.266.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.881 [info] <0.266.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.881 [info] <0.266.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.881 [info] <0.266.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.881 [info] <0.266.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.881 [info] <0.266.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.882 [info] <0.266.0> Running boot step rabbit_priority_queue defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.882 [info] <0.266.0> Priority queues enabled, real BQ is rabbit_variable_queue
annotator_message_q | 2019-08-08 09:53:22.882 [info] <0.266.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.882 [info] <0.266.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.882 [info] <0.266.0> Running boot step kernel_ready defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.882 [info] <0.266.0> Running boot step rabbit_sysmon_minder defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.882 [info] <0.266.0> Running boot step rabbit_epmd_monitor defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.884 [info] <0.266.0> Running boot step guid_generator defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.887 [info] <0.266.0> Running boot step rabbit_node_monitor defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.887 [info] <0.362.0> Starting rabbit_node_monitor
annotator_message_q | 2019-08-08 09:53:22.887 [info] <0.266.0> Running boot step delegate_sup defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.888 [info] <0.266.0> Running boot step rabbit_memory_monitor defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.888 [info] <0.266.0> Running boot step core_initialized defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.888 [info] <0.266.0> Running boot step upgrade_queues defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.900 [info] <0.266.0> Running boot step rabbit_connection_tracking defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.901 [info] <0.266.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.901 [info] <0.266.0> Running boot step rabbit_exchange_parameters defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.901 [info] <0.266.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.901 [info] <0.266.0> Running boot step rabbit_policies defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.902 [info] <0.266.0> Running boot step rabbit_policy defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.902 [info] <0.266.0> Running boot step rabbit_queue_location_validator defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.902 [info] <0.266.0> Running boot step rabbit_vhost_limit defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.902 [info] <0.266.0> Running boot step recovery defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.903 [info] <0.393.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@edc67d7ed6fc/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
annotator_message_q | 2019-08-08 09:53:22.907 [info] <0.393.0> Starting message stores for vhost '/'
annotator_message_q | 2019-08-08 09:53:22.907 [info] <0.397.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
annotator_message_q | 2019-08-08 09:53:22.908 [info] <0.393.0> Started message store of type transient for vhost '/'
annotator_message_q | 2019-08-08 09:53:22.908 [info] <0.400.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
annotator_message_q | 2019-08-08 09:53:22.909 [warning] <0.400.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
annotator_message_q | 2019-08-08 09:53:22.909 [info] <0.393.0> Started message store of type persistent for vhost '/'
annotator_message_q | 2019-08-08 09:53:22.915 [info] <0.266.0> Running boot step empty_db_check defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.915 [info] <0.266.0> Running boot step rabbit_looking_glass defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.915 [info] <0.266.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.915 [info] <0.266.0> Running boot step background_gc defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.916 [info] <0.266.0> Running boot step connection_tracking defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.916 [info] <0.266.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@edc67d7ed6fc
annotator_message_q | 2019-08-08 09:53:22.916 [info] <0.266.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@edc67d7ed6fc
annotator_message_q | 2019-08-08 09:53:22.916 [info] <0.266.0> Running boot step routing_ready defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.916 [info] <0.266.0> Running boot step pre_flight defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.916 [info] <0.266.0> Running boot step notify_cluster defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.916 [info] <0.266.0> Running boot step networking defined by app rabbit
annotator_message_q | 2019-08-08 09:53:22.917 [warning] <0.429.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
annotator_message_q | 2019-08-08 09:53:22.918 [info] <0.443.0> started TCP listener on [::]:5672
annotator_message_q | 2019-08-08 09:53:22.918 [info] <0.266.0> Running boot step direct_client defined by app rabbit
annotator_message_q |  completed with 0 plugins.
annotator_message_q | 2019-08-08 09:53:23.007 [info] <0.8.0> Server startup complete; 0 plugins started.
annotator_webclient | Using TensorFlow backend.
annotator_webclient | WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
annotator_webclient | Instructions for updating:
annotator_webclient | Colocations handled automatically by placer.
annotator_message_q | 2019-08-08 09:53:24.212 [info] <0.447.0> accepting AMQP connection <0.447.0> (172.18.0.4:48432 -> 172.18.0.3:5672)
annotator_message_q | 2019-08-08 09:53:24.214 [info] <0.447.0> connection <0.447.0> (172.18.0.4:48432 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_workers | [2019-08-08 09:53:24,214: INFO/MainProcess] Connected to amqp://user:**@messageq:5672//
annotator_message_q | 2019-08-08 09:53:24.224 [info] <0.455.0> accepting AMQP connection <0.455.0> (172.18.0.4:48434 -> 172.18.0.3:5672)
annotator_message_q | 2019-08-08 09:53:24.225 [info] <0.455.0> connection <0.455.0> (172.18.0.4:48434 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_workers | [2019-08-08 09:53:24,225: INFO/MainProcess] mingle: searching for neighbors
annotator_message_q | 2019-08-08 09:53:24.239 [info] <0.472.0> accepting AMQP connection <0.472.0> (172.18.0.4:48436 -> 172.18.0.3:5672)
annotator_message_q | 2019-08-08 09:53:24.241 [info] <0.472.0> connection <0.472.0> (172.18.0.4:48436 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_workers | [2019-08-08 09:53:25,251: INFO/MainProcess] mingle: all alone
annotator_workers | [2019-08-08 09:53:25,268: INFO/MainProcess] celery@ec319223d9ec ready.
annotator_webclient | WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py:772: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
annotator_webclient | Instructions for updating:
annotator_webclient | Use tf.cast instead.
annotator_webclient | 2019-08-08 09:53:32.064899: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2800070000 Hz
annotator_webclient | 2019-08-08 09:53:32.067661: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5566c21490e0 executing computations on platform Host. Devices:
annotator_webclient | 2019-08-08 09:53:32.067701: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
annotator_webclient | [2019-08-08 09:53:35 +0000] [10] [INFO] Loaded MaskRCNN model: /models/corrosion_latest.h5
annotator_webclient | [2019-08-08 09:53:35 +0000] [10] [WARNING] DEXTR model is disabled.
annotator_webclient | [2019-08-08 09:53:35 +0000] [10] [DEBUG] GET /socket.io/
annotator_message_q | 2019-08-08 09:53:35.090 [info] <0.509.0> accepting AMQP connection <0.509.0> (172.18.0.5:46624 -> 172.18.0.3:5672)
annotator_webclient | [2019-08-08 09:53:35 +0000] [10] [INFO] Socket connection created with admin
annotator_message_q | 2019-08-08 09:53:35.095 [info] <0.509.0> connection <0.509.0> (172.18.0.5:46624 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_webclient | [2019-08-08 09:53:35 +0000] [10] [DEBUG] GET /socket.io/
annotator_webclient | [2019-08-08 09:53:35 +0000] [10] [DEBUG] GET /socket.io/
annotator_webclient | [2019-08-08 09:55:22 +0000] [10] [DEBUG] POST /api/model/maskrcnn
annotator_webclient | [2019-08-08 09:55:26 +0000] [10] [DEBUG] POST /api/annotation/
annotator_webclient | [2019-08-08 09:55:26 +0000] [10] [INFO] admin has created an annotation for image 29
annotator_message_q | 2019-08-08 09:56:24.243 [error] <0.472.0> closing AMQP connection <0.472.0> (172.18.0.4:48436 -> 172.18.0.3:5672):
annotator_message_q | missed heartbeats from client, timeout: 60s

OS: Debian 9 (proxmox) uname -a: Linux pve 4.15.18-12-pve #1 SMP PVE 4.15.18-35 (Wed, 13 Mar 2019 08:24:42 +0100) x86_64 GNU/Linux

kevdagoat commented 4 years ago

Oh and: Docker: Docker version 19.03.1, build 74b1e89 Docker Compose: docker-compose version 1.24.1, build 4667896b

jsbroks commented 4 years ago

What do you mean its not working? I see [INFO] Loaded MaskRCNN model: /models/corrosion_latest.h5 logged to console. That is currently the only logging the maskrcnn model does.

If you would like to add more, I would accept a PR for it.

kevdagoat commented 4 years ago

I have added a custom function, copying the logging function from the init function and it doesn’t work when I press the “annotate” button

Will give you further details tonight

Sent from my iPhone

On 12 Aug 2019, at 10:12 pm, Justin Brooks notifications@github.com wrote:

What do you mean its not working? I see [INFO] Loaded MaskRCNN model: /models/corrosion_latest.h5 logged to console

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

kevdagoat commented 4 years ago

Here is a proof-of-concept mask_rcnn.py file:

from config import Config as AnnotatorConfig
from skimage.transform import resize
import imantics as im

from keras.preprocessing.image import img_to_array
from mrcnn.config import Config
import mrcnn.model as modellib
import logging
logger = logging.getLogger('gunicorn.error')

MODEL_DIR = "/workspace/models"
COCO_MODEL_PATH = AnnotatorConfig.MASK_RCNN_FILE
CLASS_NAMES = AnnotatorConfig.MASK_RCNN_CLASSES.split(',')

class CocoConfig(Config):
    """
    Configuration for COCO Dataset.
    """
    NAME = "coco"
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1
    NUM_CLASSES = len(CLASS_NAMES)

class MaskRCNN():

    def __init__(self):

        self.config = CocoConfig()
        self.model = modellib.MaskRCNN(
            mode="inference",
            model_dir=MODEL_DIR,
            config=self.config
        )
        try:
            self.model.load_weights(COCO_MODEL_PATH, by_name=True)
            self.model.keras_model._make_predict_function()
            logger.info(f"Loaded MaskRCNN model: {COCO_MODEL_PATH}")
        except:
            logger.error(f"Could not load MaskRCNN model (place 'mask_rcnn_coco.h5' in the models directory)")
            self.model = None

    def detect(self, image):
        logger.info(f"Starting MaskRCNN Annotation on selected image.")
        if self.model is None:
            return {}

        image = image.convert('RGB')
        width, height = image.size
        image.thumbnail((1024, 1024))

        image = img_to_array(image)
        result = self.model.detect([image])[0]

        masks = result.get('masks')
        class_ids = result.get('class_ids')

        coco_image = im.Image(width=width, height=height)

        for i in range(masks.shape[-1]):
            mask = resize(masks[..., i], (height, width))
            mask = im.Mask(mask)
            class_id = class_ids[i]
            class_name = CLASS_NAMES[class_id]
            category = im.Category(class_name)
            coco_image.add(mask, category=category)
        logger.info(f"Annotated image")
        return coco_image.coco()

model = MaskRCNN()

I have also tried self.logger.info()

Console log (Model is used at the end):

Starting annotator_mongodb   ... done
Starting annotator_message_q ... done
Starting annotator_workers   ... done
Starting annotator_webclient ... done
Attaching to annotator_message_q, annotator_mongodb, annotator_workers, annotator_webclient
annotator_workers | fatal: Not a git repository (or any of the parent directories): .git
annotator_webclient | [2019-08-14 09:21:02 +0000] [6] [DEBUG] Current configuration:
annotator_webclient |   config: webserver/gunicorn_config.py
annotator_webclient |   bind: ['0.0.0.0:5000']
annotator_webclient |   backlog: 2048
annotator_webclient |   workers: 1
annotator_webclient |   worker_class: eventlet
annotator_webclient |   threads: 1
annotator_webclient |   worker_connections: 1000
annotator_webclient |   max_requests: 0
annotator_webclient |   max_requests_jitter: 0
annotator_webclient |   timeout: 180
annotator_webclient |   graceful_timeout: 30
annotator_webclient |   keepalive: 2
annotator_webclient |   limit_request_line: 4094
annotator_webclient |   limit_request_fields: 100
annotator_webclient |   limit_request_field_size: 8190
annotator_webclient |   reload: False
annotator_webclient |   reload_engine: auto
annotator_webclient |   reload_extra_files: []
annotator_webclient |   spew: False
annotator_webclient |   check_config: False
annotator_webclient |   preload_app: False
annotator_webclient |   sendfile: False
annotator_webclient |   reuse_port: False
annotator_webclient |   chdir: /workspace
annotator_webclient |   daemon: False
annotator_webclient |   raw_env: []
annotator_webclient |   pidfile: None
annotator_webclient |   worker_tmp_dir: None
annotator_webclient |   user: 0
annotator_webclient |   group: 0
annotator_webclient |   umask: 0
annotator_webclient |   initgroups: False
annotator_webclient |   tmp_upload_dir: None
annotator_webclient |   secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
annotator_webclient |   forwarded_allow_ips: ['127.0.0.1']
annotator_webclient |   accesslog: None
annotator_webclient |   disable_redirect_access_to_syslog: False
annotator_webclient |   access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
annotator_webclient |   errorlog: -
annotator_webclient |   loglevel: debug
annotator_webclient |   capture_output: False
annotator_webclient |   logger_class: gunicorn.glogging.Logger
annotator_webclient |   logconfig: None
annotator_webclient |   logconfig_dict: {}
annotator_webclient |   syslog_addr: udp://localhost:514
annotator_webclient |   syslog: False
annotator_webclient |   syslog_prefix: None
annotator_webclient |   syslog_facility: user
annotator_webclient |   enable_stdio_inheritance: False
annotator_webclient |   statsd_host: None
annotator_webclient |   statsd_prefix:
annotator_webclient |   proc_name: None
annotator_webclient |   default_proc_name: webserver:app
annotator_webclient |   pythonpath: None
annotator_webclient |   paste: None
annotator_webclient |   on_starting: <function OnStarting.on_starting at 0x7f48867fe2f0>
annotator_webclient |   on_reload: <function OnReload.on_reload at 0x7f48867fe400>
annotator_webclient |   when_ready: <function WhenReady.when_ready at 0x7f48867fe510>
annotator_webclient |   pre_fork: <function Prefork.pre_fork at 0x7f48867fe620>
annotator_webclient |   post_fork: <function Postfork.post_fork at 0x7f48867fe730>
annotator_webclient |   post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f48867fe840>
annotator_webclient |   worker_int: <function WorkerInt.worker_int at 0x7f48867fe950>
annotator_webclient |   worker_abort: <function WorkerAbort.worker_abort at 0x7f48867fea60>
annotator_webclient |   pre_exec: <function PreExec.pre_exec at 0x7f48867feb70>
annotator_webclient |   pre_request: <function PreRequest.pre_request at 0x7f48867fec80>
annotator_webclient |   post_request: <function PostRequest.post_request at 0x7f48867fed08>
annotator_webclient |   child_exit: <function ChildExit.child_exit at 0x7f48867fee18>
annotator_webclient |   worker_exit: <function WorkerExit.worker_exit at 0x7f48867fef28>
annotator_webclient |   nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f488657b0d0>
annotator_webclient |   on_exit: <function OnExit.on_exit at 0x7f488657b1e0>
annotator_webclient |   proxy_protocol: False
annotator_webclient |   proxy_allow_ips: ['127.0.0.1']
annotator_webclient |   keyfile: None
annotator_webclient |   certfile: None
annotator_webclient |   ssl_version: 2
annotator_webclient |   cert_reqs: 0
annotator_webclient |   ca_certs: None
annotator_webclient |   suppress_ragged_eofs: True
annotator_webclient |   do_handshake_on_connect: False
annotator_webclient |   ciphers: TLSv1
annotator_webclient |   raw_paste_global_conf: []
annotator_webclient | [2019-08-14 09:21:02 +0000] [6] [INFO] Starting gunicorn 19.9.0
annotator_webclient | [2019-08-14 09:21:02 +0000] [6] [DEBUG] Arbiter booted
annotator_webclient | [2019-08-14 09:21:02 +0000] [6] [INFO] Listening at: http://0.0.0.0:5000 (6)
annotator_webclient | [2019-08-14 09:21:02 +0000] [6] [INFO] Using worker: eventlet
annotator_webclient | [2019-08-14 09:21:02 +0000] [10] [INFO] Booting worker with pid: 10
annotator_webclient | [2019-08-14 09:21:02 +0000] [6] [DEBUG] 1 workers
annotator_workers | /usr/local/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
annotator_workers | absolutely not recommended!
annotator_workers |
annotator_workers | Please specify a different user using the --uid option.
annotator_workers |
annotator_workers | User information: uid=0 euid=0 gid=0 egid=0
annotator_workers |
annotator_workers |   uid=uid, euid=euid, gid=gid, egid=egid,
annotator_workers | [2019-08-14 09:21:03,695: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: [Errno 111] Connection refused.
annotator_workers | Trying again in 2.00 seconds...
annotator_workers |
annotator_message_q | 2019-08-14 09:21:04.433 [info] <0.8.0> Feature flags: list of feature flags found:
annotator_message_q | 2019-08-14 09:21:04.433 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
annotator_message_q | 2019-08-14 09:21:04.502 [info] <0.266.0>
annotator_message_q |  Starting RabbitMQ 3.7.17 on Erlang 22.0.7
annotator_message_q |  Copyright (C) 2007-2019 Pivotal Software, Inc.
annotator_message_q |  Licensed under the MPL.  See https://www.rabbitmq.com/
annotator_message_q |
annotator_message_q |   ##  ##
annotator_message_q |   ##  ##      RabbitMQ 3.7.17. Copyright (C) 2007-2019 Pivotal Software, Inc.
annotator_message_q |   ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
annotator_message_q |   ######  ##
annotator_message_q |   ##########  Logs: <stdout>
annotator_message_q |
annotator_message_q |               Starting broker...
annotator_message_q | 2019-08-14 09:21:04.503 [info] <0.266.0>
annotator_message_q |  node           : rabbit@08deb265bff6
annotator_message_q |  home dir       : /var/lib/rabbitmq
annotator_message_q |  config file(s) : /etc/rabbitmq/rabbitmq.conf
annotator_message_q |  cookie hash    : dUJut5WwLn6HOxdneFWT5g==
annotator_message_q |  log(s)         : <stdout>
annotator_message_q |  database dir   : /var/lib/rabbitmq/mnesia/rabbit@08deb265bff6
annotator_message_q | 2019-08-14 09:21:04.511 [info] <0.266.0> Running boot step pre_boot defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.511 [info] <0.266.0> Running boot step rabbit_core_metrics defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.513 [info] <0.266.0> Running boot step rabbit_alarm defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.516 [info] <0.293.0> Memory high watermark set to 25779 MiB (27031968153 bytes) of 64449 MiB (67579920384 bytes) total
annotator_message_q | 2019-08-14 09:21:04.520 [info] <0.295.0> Enabling free disk space monitoring
annotator_message_q | 2019-08-14 09:21:04.520 [info] <0.295.0> Disk free limit set to 50MB
annotator_message_q | 2019-08-14 09:21:04.522 [info] <0.266.0> Running boot step code_server_cache defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.522 [info] <0.266.0> Running boot step file_handle_cache defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.523 [info] <0.298.0> Limiting to approx 1048476 file handles (943626 sockets)
annotator_message_q | 2019-08-14 09:21:04.523 [info] <0.299.0> FHC read buffering:  OFF
annotator_message_q | 2019-08-14 09:21:04.523 [info] <0.299.0> FHC write buffering: ON
annotator_message_q | 2019-08-14 09:21:04.523 [info] <0.266.0> Running boot step worker_pool defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.523 [info] <0.269.0> Will use 40 processes for default worker pool
annotator_message_q | 2019-08-14 09:21:04.523 [info] <0.269.0> Starting worker pool 'worker_pool' with 40 processes in it
annotator_message_q | 2019-08-14 09:21:04.525 [info] <0.266.0> Running boot step database defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.526 [info] <0.266.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q | 2019-08-14 09:21:04.560 [info] <0.266.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q | 2019-08-14 09:21:04.560 [info] <0.266.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
annotator_message_q | 2019-08-14 09:21:04.560 [info] <0.266.0> Running boot step database_sync defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.560 [info] <0.266.0> Running boot step feature_flags defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.560 [info] <0.266.0> Running boot step codec_correctness_check defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.560 [info] <0.266.0> Running boot step external_infrastructure defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.560 [info] <0.266.0> Running boot step rabbit_registry defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_queue_location_random defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_event defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.561 [info] <0.266.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.562 [info] <0.266.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.562 [info] <0.266.0> Running boot step rabbit_priority_queue defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.562 [info] <0.266.0> Priority queues enabled, real BQ is rabbit_variable_queue
annotator_message_q | 2019-08-14 09:21:04.562 [info] <0.266.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.562 [info] <0.266.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.562 [info] <0.266.0> Running boot step kernel_ready defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.562 [info] <0.266.0> Running boot step rabbit_sysmon_minder defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.562 [info] <0.266.0> Running boot step rabbit_epmd_monitor defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.565 [info] <0.266.0> Running boot step guid_generator defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.567 [info] <0.266.0> Running boot step rabbit_node_monitor defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.567 [info] <0.362.0> Starting rabbit_node_monitor
annotator_message_q | 2019-08-14 09:21:04.568 [info] <0.266.0> Running boot step delegate_sup defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.568 [info] <0.266.0> Running boot step rabbit_memory_monitor defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.568 [info] <0.266.0> Running boot step core_initialized defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.568 [info] <0.266.0> Running boot step upgrade_queues defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.581 [info] <0.266.0> Running boot step rabbit_connection_tracking defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.581 [info] <0.266.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.581 [info] <0.266.0> Running boot step rabbit_exchange_parameters defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.581 [info] <0.266.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.582 [info] <0.266.0> Running boot step rabbit_policies defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.582 [info] <0.266.0> Running boot step rabbit_policy defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.582 [info] <0.266.0> Running boot step rabbit_queue_location_validator defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.582 [info] <0.266.0> Running boot step rabbit_vhost_limit defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.583 [info] <0.266.0> Running boot step recovery defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.584 [info] <0.393.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@08deb265bff6/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
annotator_message_q | 2019-08-14 09:21:04.588 [info] <0.393.0> Starting message stores for vhost '/'
annotator_message_q | 2019-08-14 09:21:04.588 [info] <0.397.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
annotator_message_q | 2019-08-14 09:21:04.589 [info] <0.393.0> Started message store of type transient for vhost '/'
annotator_message_q | 2019-08-14 09:21:04.589 [info] <0.400.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
annotator_message_q | 2019-08-14 09:21:04.589 [warning] <0.400.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
annotator_message_q | 2019-08-14 09:21:04.590 [info] <0.393.0> Started message store of type persistent for vhost '/'
annotator_message_q | 2019-08-14 09:21:04.596 [info] <0.266.0> Running boot step empty_db_check defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.596 [info] <0.266.0> Running boot step rabbit_looking_glass defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.596 [info] <0.266.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.596 [info] <0.266.0> Running boot step background_gc defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.596 [info] <0.266.0> Running boot step connection_tracking defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.596 [info] <0.266.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@08deb265bff6
annotator_message_q | 2019-08-14 09:21:04.597 [info] <0.266.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@08deb265bff6
annotator_message_q | 2019-08-14 09:21:04.597 [info] <0.266.0> Running boot step routing_ready defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.597 [info] <0.266.0> Running boot step pre_flight defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.597 [info] <0.266.0> Running boot step notify_cluster defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.597 [info] <0.266.0> Running boot step networking defined by app rabbit
annotator_message_q | 2019-08-14 09:21:04.598 [warning] <0.429.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
annotator_message_q | 2019-08-14 09:21:04.598 [info] <0.443.0> started TCP listener on [::]:5672
annotator_message_q | 2019-08-14 09:21:04.599 [info] <0.266.0> Running boot step direct_client defined by app rabbit
annotator_message_q |  completed with 0 plugins.
annotator_message_q | 2019-08-14 09:21:04.693 [info] <0.8.0> Server startup complete; 0 plugins started.
annotator_webclient | Using TensorFlow backend.
annotator_webclient | WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
annotator_webclient | Instructions for updating:
annotator_webclient | Colocations handled automatically by placer.
annotator_message_q | 2019-08-14 09:21:05.705 [info] <0.447.0> accepting AMQP connection <0.447.0> (172.18.0.4:40232 -> 172.18.0.3:5672)
annotator_message_q | 2019-08-14 09:21:05.707 [info] <0.447.0> connection <0.447.0> (172.18.0.4:40232 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_workers | [2019-08-14 09:21:05,708: INFO/MainProcess] Connected to amqp://user:**@messageq:5672//
annotator_message_q | 2019-08-14 09:21:05.718 [info] <0.455.0> accepting AMQP connection <0.455.0> (172.18.0.4:40234 -> 172.18.0.3:5672)
annotator_message_q | 2019-08-14 09:21:05.720 [info] <0.455.0> connection <0.455.0> (172.18.0.4:40234 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_workers | [2019-08-14 09:21:05,720: INFO/MainProcess] mingle: searching for neighbors
annotator_message_q | 2019-08-14 09:21:05.735 [info] <0.472.0> accepting AMQP connection <0.472.0> (172.18.0.4:40236 -> 172.18.0.3:5672)
annotator_message_q | 2019-08-14 09:21:05.737 [info] <0.472.0> connection <0.472.0> (172.18.0.4:40236 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_workers | [2019-08-14 09:21:06,746: INFO/MainProcess] mingle: all alone
annotator_workers | [2019-08-14 09:21:06,762: INFO/MainProcess] celery@79ae66ce15bc ready.
annotator_webclient | WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py:772: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
annotator_webclient | Instructions for updating:
annotator_webclient | Use tf.cast instead.
annotator_webclient | 2019-08-14 09:21:13.772965: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2800070000 Hz
annotator_webclient | 2019-08-14 09:21:13.775273: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5567478756d0 executing computations on platform Host. Devices:
annotator_webclient | 2019-08-14 09:21:13.775303: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
annotator_webclient | [2019-08-14 09:21:16 +0000] [10] [INFO] Loaded MaskRCNN model: /models/corrosion_latest.h5
annotator_webclient | [2019-08-14 09:21:16 +0000] [10] [WARNING] DEXTR model is disabled.
annotator_webclient | [2019-08-14 09:21:16 +0000] [10] [DEBUG] GET /socket.io/
annotator_message_q | 2019-08-14 09:21:16.910 [info] <0.509.0> accepting AMQP connection <0.509.0> (172.18.0.5:38424 -> 172.18.0.3:5672)
annotator_webclient | [2019-08-14 09:21:16 +0000] [10] [INFO] Socket connection created with admin
annotator_message_q | 2019-08-14 09:21:16.914 [info] <0.509.0> connection <0.509.0> (172.18.0.5:38424 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_webclient | [2019-08-14 09:21:16 +0000] [10] [DEBUG] GET /socket.io/
annotator_webclient | [2019-08-14 09:21:16 +0000] [10] [DEBUG] Closing connection.
annotator_webclient | [2019-08-14 09:21:16 +0000] [10] [DEBUG] GET /socket.io/
annotator_webclient | [2019-08-14 09:21:23 +0000] [10] [DEBUG] POST /api/model/maskrcnn
annotator_webclient | [2019-08-14 09:21:27 +0000] [10] [DEBUG] POST /api/annotation/
annotator_webclient | [2019-08-14 09:21:27 +0000] [10] [DEBUG] POST /api/annotation/
annotator_webclient | [2019-08-14 09:21:27 +0000] [10] [INFO] admin has created an annotation for image 22
annotator_webclient | [2019-08-14 09:21:27 +0000] [10] [INFO] admin has created an annotation for image 22