trackit / trackit2-home

TrackIt helps you to optimize your AWS cloud
https://trackit.io/
Apache License 2.0
48 stars 8 forks source link

Unable to register in trackit #69

Open sathyajackson opened 5 years ago

sathyajackson commented 5 years ago

I have set up the complete UI using the docker and its up and running. Now that I am trying to register the same but unable to.

when I try to register, I get an error as "An error has occurred".

Not sure what wrong can someone help me out to get the UI up and running.

thibautcornolti commented 5 years ago

Hi @sathyajackson

Can I have some logs? You can get them by typing docker-compose logs -f.

Thanks.

lfrancois commented 5 years ago

@sathyajackson can we setup a call with you later this week or next week? So that we can troubleshoot the issue with you?

sathyajackson commented 5 years ago

Sorry sir, if your planning to make me buy then I am not the right person for you talk....

Even the company I work are not ready to buy paid software sorry........

sathyajackson commented 5 years ago

Not my fault......peace

lfrancois commented 5 years ago

No worries :) We just want to understand the issue and get constructive feedback about what we are building. You can reach me at ludo@trackit.io and we will setup a troubleshooting session. Also you can test the free version on https://trackit.io. Nobody will ask you any credit card or anything :)

sathyajackson commented 5 years ago

Here is the complete log file

WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string. WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string. WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string. WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string. Attaching to trackit_api_1_d0ea38fddddd, trackit_es_1_c00afa428f94, trackit_sql_1_baad39f9a608, trackit_ui_1_c7e00a815717 api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:32.478736607Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.3:3306: getsockopt: connection refused"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:34.479610691Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.3:3306: getsockopt: connection refused"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:36.480478594Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.3:3306: getsockopt: connection refused"} es_1_c00afa428f94 | [2018-12-06T14:43:33,114][INFO ][o.e.n.Node ] [] initializing ... sql_1_baad39f9a608 | Initializing database es_1_c00afa428f94 | [2018-12-06T14:43:33,166][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] using [1] data paths, mounts [[/ (overlay)]], net usable_space [8.4gb], net total_space [29.4gb], spins? [possibly], types [overlay] sql_1_baad39f9a608 | sql_1_baad39f9a608 | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER ! api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:38.481242486Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.3:3306: getsockopt: connection refused"} sql_1_baad39f9a608 | To do so, start the server, then issue the following commands: es_1_c00afa428f94 | [2018-12-06T14:43:33,166][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] heap size [1.9gb], compressed ordinary object pointers [true] api_1_d0ea38fddddd | {"level":"info","time":"2018-12-06T14:43:40.48230592Z","message":"Successfully connected to SQL database."} ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. sql_1_baad39f9a608 | es_1_c00afa428f94 | [2018-12-06T14:43:33,167][INFO ][o.e.n.Node ] node name [Izt4zdP] derived from node ID [Izt4zdPRTW2IHvngLppy2w]; set [node.name] to override api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:46.249540739Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching ' sql_1_baad39f9a608 | '/usr/bin/mysqladmin' -u root password 'new-password' es_1_c00afa428f94 | [2018-12-06T14:43:33,168][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:53.888146148Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | '/usr/bin/mysqladmin' -u root -h password 'new-password' ui_1_c7e00a815717 | 2018-12-06 14:44:48,247 CRIT Supervisor running as root (no user in config file) es_1_c00afa428f94 | [2018-12-06T14:43:33,168][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:01.356401632Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | ui_1_c7e00a815717 | 2018-12-06 14:44:48,248 INFO supervisord started with pid 1 es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [aggs-matrix-stats] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:08.966041141Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | Alternatively you can run: ui_1_c7e00a815717 | 2018-12-06 14:44:49,250 INFO spawned: 'webui' with pid 10 es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [ingest-common] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:16.532639788Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | '/usr/bin/mysql_secure_installation' ui_1_c7e00a815717 | 2018-12-06 14:44:50,428 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-expression] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:24.19991604Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | ui_1_c7e00a815717 | 2018-12-06 15:30:52,465 WARN received SIGTERM indicating exit request es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-groovy] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:31.702865478Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | which will also give you the option of removing the test ui_1_c7e00a815717 | 2018-12-06 15:30:52,466 INFO waiting for webui to die api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:39.203860775Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-mustache] sql_1_baad39f9a608 | databases and anonymous user created by default. This is ui_1_c7e00a815717 | 2018-12-06 15:30:52,467 INFO stopped: webui (terminated by SIGTERM) es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-painless] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:46.748528399Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | strongly recommended for production servers. ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [parent-join] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:53.752770785Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"} sql_1_baad39f9a608 | ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching ' es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [percolator] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:00.947007596Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | See the MariaDB Knowledgebase at http://mariadb.com/kb or the ui_1_c7e00a815717 | 2018-12-06 15:32:40,828 CRIT Supervisor running as root (no user in config file) es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [reindex] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:08.372114485Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | MySQL manual for more instructions. ui_1_c7e00a815717 | 2018-12-06 15:32:40,830 INFO supervisord started with pid 1 es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty3] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:15.532798053Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | ui_1_c7e00a815717 | 2018-12-06 15:32:41,831 INFO spawned: 'webui' with pid 10 es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty4] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:22.812170558Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | Please report any problems at http://mariadb.org/jira ui_1_c7e00a815717 | 2018-12-06 15:32:43,025 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) es_1_c00afa428f94 | [2018-12-06T14:43:34,348][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-geoip] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:30.300248305Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | ui_1_c7e00a815717 | 2018-12-07 13:42:52,609 WARN received SIGTERM indicating exit request es_1_c00afa428f94 | [2018-12-06T14:43:34,348][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-user-agent] api_1_d0ea38fddddd | {"level":"error","time":"2018-12-06T14:45:32.300391772Z","message":"Failed to connect to ElasticSearch database. Not retrying."} sql_1_baad39f9a608 | The latest information about MariaDB is available at http://mariadb.org/. es_1_c00afa428f94 | [2018-12-06T14:43:34,348][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [x-pack] api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference es_1_c00afa428f94 | [2018-12-06T14:43:37,156][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/56] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV sql_1_baad39f9a608 | You can find additional information about the MySQL part at: ui_1_c7e00a815717 | 2018-12-07 13:42:52,610 INFO waiting for webui to die es_1_c00afa428f94 | [2018-12-06T14:43:37,188][INFO ][o.e.d.DiscoveryModule ] [Izt4zdP] using discovery type [zen] api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33] sql_1_baad39f9a608 | http://dev.mysql.com ui_1_c7e00a815717 | 2018-12-07 13:42:52,611 INFO stopped: webui (terminated by SIGTERM) es_1_c00afa428f94 | [2018-12-06T14:43:37,800][INFO ][o.e.n.Node ] initialized api_1_d0ea38fddddd | sql_1_baad39f9a608 | Consider joining MariaDB's strong and vibrant community: ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. es_1_c00afa428f94 | [2018-12-06T14:43:37,800][INFO ][o.e.n.Node ] [Izt4zdP] starting ... api_1_d0ea38fddddd | goroutine 1 [running]: sql_1_baad39f9a608 | https://mariadb.org/get-involved/ ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching ' es_1_c00afa428f94 | [2018-12-06T14:43:37,993][INFO ][o.e.t.TransportService ] [Izt4zdP] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300} api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4204c5980, 0x108a829, 0x3, 0xc4204ab9e0, 0x14, 0xc42054a0c0, 0xe06aa0, 0xc4203f6dd0, ...) sql_1_baad39f9a608 | ui_1_c7e00a815717 | 2018-12-07 13:44:41,637 CRIT Supervisor running as root (no user in config file) es_1_c00afa428f94 | [2018-12-06T14:43:38,001][INFO ][o.e.b.BootstrapChecks ] [Izt4zdP] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks sql_1_baad39f9a608 | Database initialized api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3 ui_1_c7e00a815717 | 2018-12-07 13:44:41,639 INFO supervisord started with pid 1 es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed sql_1_baad39f9a608 | MySQL init process in progress... api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequest(0x0, 0x189d380, 0xc4204c5980, 0x108a829, 0x3, 0xc4204ab9e0, 0x14, 0xc42054a0c0, 0xe06aa0, 0xc4203f6dd0, ...) ui_1_c7e00a815717 | 2018-12-07 13:44:42,641 INFO spawned: 'webui' with pid 9 es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 102 ... api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f es_1_c00afa428f94 | [2018-12-06T14:43:38,009][INFO ][o.e.n.Node ] [Izt4zdP] stopping ... ui_1_c7e00a815717 | 2018-12-07 13:44:43,845 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc42056fe38, 0x189d380, 0xc4204c5980, 0x189d380, 0xc4204c5980, 0xc4203f6d90) es_1_c00afa428f94 | [2018-12-06T14:43:38,115][INFO ][o.e.n.Node ] [Izt4zdP] stopped ui_1_c7e00a815717 | 2018-12-07 13:49:10,098 WARN received SIGTERM indicating exit request sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Uses event mutexes api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111 es_1_c00afa428f94 | [2018-12-06T14:43:38,115][INFO ][o.e.n.Node ] [Izt4zdP] closing ... ui_1_c7e00a815717 | 2018-12-07 13:49:10,098 INFO waiting for webui to die sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Compressed tables use zlib 1.2.8 api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1() es_1_c00afa428f94 | [2018-12-06T14:43:38,123][INFO ][o.e.n.Node ] [Izt4zdP] closed ui_1_c7e00a815717 | 2018-12-07 13:49:10,100 INFO stopped: webui (terminated by SIGTERM) sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Using Linux native AIO api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea es_1_c00afa428f94 | [2018-12-06T14:43:38,126][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Number of pools: 1 api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init() es_1_c00afa428f94 | [2018-12-06T14:44:49,802][INFO ][o.e.n.Node ] [] initializing ... ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching ' sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Using SSE2 crc32 instructions api_1_d0ea38fddddd | :1 +0x183 es_1_c00afa428f94 | [2018-12-06T14:44:49,861][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] using [1] data paths, mounts [[/ (overlay)]], net usable_space [7.9gb], net total_space [29.4gb], spins? [possibly], types [overlay] ui_1_c7e00a815717 | 2018-12-07 13:49:11,308 CRIT Supervisor running as root (no user in config file) sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init() es_1_c00afa428f94 | [2018-12-06T14:44:49,861][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] heap size [1.9gb], compressed ordinary object pointers [true] ui_1_c7e00a815717 | 2018-12-07 13:49:11,309 INFO supervisord started with pid 1 sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Completed initialization of buffer pool es_1_c00afa428f94 | [2018-12-06T14:44:49,863][INFO ][o.e.n.Node ] node name [Izt4zdP] derived from node ID [Izt4zdPRTW2IHvngLppy2w]; set [node.name] to override api_1_d0ea38fddddd | :1 +0x8c ui_1_c7e00a815717 | 2018-12-07 13:49:12,311 INFO spawned: 'webui' with pid 10 sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064142829312 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). es_1_c00afa428f94 | [2018-12-06T14:44:49,863][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16] api_1_d0ea38fddddd | main.init() ui_1_c7e00a815717 | 2018-12-07 13:49:13,446 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Highest supported file format is Barracuda. es_1_c00afa428f94 | [2018-12-06T14:44:49,863][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch] api_1_d0ea38fddddd | :1 +0x78 sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: 128 out of 128 rollback segments are active. es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [aggs-matrix-stats] api_1_d0ea38fddddd | {"level":"info","time":"2018-12-06T15:32:41.979048074Z","message":"Successfully connected to SQL database."} es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [ingest-common] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:32:47.083580545Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Creating shared tablespace for temporary tables es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-expression] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:32:54.258577463Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-groovy] sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: File './ibtmp1' size is now 12 MB. api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:01.683415811Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-mustache] sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Waiting for purge to start es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-painless] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:08.828629445Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: 5.7.19 started; log sequence number 1619987 es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [parent-join] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:16.423000405Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:34 140063765681920 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [percolator] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:23.733119339Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:34 140063765681920 [Note] InnoDB: Buffer pool(s) load completed at 181206 14:43:34 es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [reindex] sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] Plugin 'FEEDBACK' is disabled. es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty3] es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty4] sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Warning] 'user' entry 'root@397e1235ec88' ignored in --skip-name-resolve mode. api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:30.919414391Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-geoip] sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode. api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:38.10388314Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-user-agent] sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] Reading of all Master_info entries succeded api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:45.418916162Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [x-pack] sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] Added new Master_info '' to hash table api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:52.590513122Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T14:44:53,377][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] mysqld: ready for connections. api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:59.853203755Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T14:44:53,412][INFO ][o.e.d.DiscoveryModule ] [Izt4zdP] using discovery type [zen] sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+maria~jessie' socket: '/var/run/mysqld/mysqld.sock' port: 0 mariadb.org binary distribution api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:34:06.933498739Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T14:44:54,034][INFO ][o.e.n.Node ] initialized sql_1_baad39f9a608 | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it. api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:34:14.122299203Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | GENERATED ROOT PASSWORD: Uengoo6zoov4loof0ji2hies1ga0ain9 es_1_c00afa428f94 | [2018-12-06T14:44:54,035][INFO ][o.e.n.Node ] [Izt4zdP] starting ... api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:34:21.199974774Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:36 140064722687744 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode. api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:34:28.422354273Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:36 140064722687744 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode. es_1_c00afa428f94 | [2018-12-06T14:44:54,225][INFO ][o.e.t.TransportService ] [Izt4zdP] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300} api_1_d0ea38fddddd | {"level":"error","time":"2018-12-06T15:34:30.42248925Z","message":"Failed to connect to ElasticSearch database. Not retrying."} sql_1_baad39f9a608 | es_1_c00afa428f94 | [2018-12-06T14:44:54,233][INFO ][o.e.b.BootstrapChecks ] [Izt4zdP] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference sql_1_baad39f9a608 | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/schema.sql es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33] sql_1_baad39f9a608 | es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] sql_1_baad39f9a608 | es_1_c00afa428f94 | [2018-12-06T14:44:54,241][INFO ][o.e.n.Node ] [Izt4zdP] stopping ... sql_1_baad39f9a608 | 2018-12-06 14:43:37 140064722384640 [Note] mysqld (unknown): Normal shutdown api_1_d0ea38fddddd | es_1_c00afa428f94 | [2018-12-06T14:44:54,348][INFO ][o.e.n.Node ] [Izt4zdP] stopped sql_1_baad39f9a608 | api_1_d0ea38fddddd | goroutine 1 [running]: es_1_c00afa428f94 | [2018-12-06T14:44:54,349][INFO ][o.e.n.Node ] [Izt4zdP] closing ... sql_1_baad39f9a608 | 2018-12-06 14:43:37 140064722384640 [Note] Event Scheduler: Purging the queue. 0 events api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4205d41e0, 0x108a829, 0x3, 0xc420503a80, 0x14, 0xc4205abe90, 0xe06aa0, 0xc42042ac80, ...) es_1_c00afa428f94 | [2018-12-06T14:44:54,357][INFO ][o.e.n.Node ] [Izt4zdP] closed sql_1_baad39f9a608 | 2018-12-06 14:43:37 140064017331968 [Note] InnoDB: FTS optimize thread exiting. api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3 es_1_c00afa428f94 | [2018-12-06T15:32:42,634][INFO ][o.e.n.Node ] [] initializing ... sql_1_baad39f9a608 | 2018-12-06 14:43:37 140064722384640 [Note] InnoDB: Starting shutdown... api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequest(0x0, 0x189d380, 0xc4205d41e0, 0x108a829, 0x3, 0xc420503a80, 0x14, 0xc4205abe90, 0xe06aa0, 0xc42042ac80, ...) es_1_c00afa428f94 | [2018-12-06T15:32:42,689][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] using [1] data paths, mounts [[/ (overlay)]], net usable_space [7.6gb], net total_space [29.4gb], spins? [possibly], types [overlay] sql_1_baad39f9a608 | 2018-12-06 14:43:37 140063765681920 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f sql_1_baad39f9a608 | 2018-12-06 14:43:37 140063765681920 [Note] InnoDB: Buffer pool(s) dump completed at 181206 14:43:37 api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc420601e38, 0x189d380, 0xc4205d41e0, 0x189d380, 0xc4205d41e0, 0xc42042ac40) sql_1_baad39f9a608 | 2018-12-06 14:43:38 140064722384640 [Note] InnoDB: Shutdown completed; log sequence number 2179604 api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111 es_1_c00afa428f94 | [2018-12-06T15:32:42,689][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] heap size [1.9gb], compressed ordinary object pointers [true] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140064722384640 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1() es_1_c00afa428f94 | [2018-12-06T15:32:42,690][INFO ][o.e.n.Node ] node name [Izt4zdP] derived from node ID [Izt4zdPRTW2IHvngLppy2w]; set [node.name] to override sql_1_baad39f9a608 | 2018-12-06 14:43:38 140064722384640 [Note] mysqld: Shutdown complete api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea sql_1_baad39f9a608 | es_1_c00afa428f94 | [2018-12-06T15:32:42,691][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16] es_1_c00afa428f94 | [2018-12-06T15:32:42,691][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch] es_1_c00afa428f94 | [2018-12-06T15:32:43,741][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [aggs-matrix-stats] api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init() es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [ingest-common] api_1_d0ea38fddddd | :1 +0x183 api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init() sql_1_baad39f9a608 | es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-expression] api_1_d0ea38fddddd | :1 +0x8c sql_1_baad39f9a608 | MySQL init process done. Ready for start up. es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-groovy] api_1_d0ea38fddddd | main.init() es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-mustache] api_1_d0ea38fddddd | :1 +0x78 es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-painless] api_1_d0ea38fddddd | {"level":"info","time":"2018-12-07T13:44:42.762834853Z","message":"Successfully connected to SQL database."} sql_1_baad39f9a608 | es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [parent-join] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:44:47.855187737Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 1 ... es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [percolator] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:44:55.204535439Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [reindex] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Uses event mutexes api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:02.530741589Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty3] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Compressed tables use zlib 1.2.8 api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:10.156413673Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty4] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Using Linux native AIO es_1_c00afa428f94 | [2018-12-06T15:32:43,743][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-geoip] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Number of pools: 1 es_1_c00afa428f94 | [2018-12-06T15:32:43,743][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-user-agent] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:17.543638008Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Using SSE2 crc32 instructions es_1_c00afa428f94 | [2018-12-06T15:32:43,743][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [x-pack] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:25.198023378Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M es_1_c00afa428f94 | [2018-12-06T15:32:46,221][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/56] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:32.683421179Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Completed initialization of buffer pool es_1_c00afa428f94 | [2018-12-06T15:32:46,247][INFO ][o.e.d.DiscoveryModule ] [Izt4zdP] using discovery type [zen] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:40.051306241Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-06 14:43:38 140549769058048 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). es_1_c00afa428f94 | [2018-12-06T15:32:46,746][INFO ][o.e.n.Node ] initialized api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:47.468262623Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T15:32:46,746][INFO ][o.e.n.Node ] [Izt4zdP] starting ... sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Highest supported file format is Barracuda. api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:54.697136828Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T15:32:46,912][INFO ][o.e.t.TransportService ] [Izt4zdP] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300} sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: 128 out of 128 rollback segments are active. api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:01.864115713Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T15:32:46,918][INFO ][o.e.b.BootstrapChecks ] [Izt4zdP] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Creating shared tablespace for temporary tables api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:09.181044801Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:16.538487676Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: File './ibtmp1' size is now 12 MB. api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:23.730707588Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T15:32:46,926][INFO ][o.e.n.Node ] [Izt4zdP] stopping ... sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: 5.7.19 started; log sequence number 2179604 api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:31.146089002Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-06T15:32:47,019][INFO ][o.e.n.Node ] [Izt4zdP] stopped sql_1_baad39f9a608 | 2018-12-06 14:43:38 140549600286464 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool api_1_d0ea38fddddd | {"level":"error","time":"2018-12-07T13:46:33.146250193Z","message":"Failed to connect to ElasticSearch database. Not retrying."} es_1_c00afa428f94 | [2018-12-06T15:32:47,019][INFO ][o.e.n.Node ] [Izt4zdP] closing ... sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] Plugin 'FEEDBACK' is disabled. api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference es_1_c00afa428f94 | [2018-12-06T15:32:47,027][INFO ][o.e.n.Node ] [Izt4zdP] closed sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] Server socket created on IP: '::'. api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33] es_1_c00afa428f94 | [2018-12-06T15:32:47,028][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode. api_1_d0ea38fddddd | es_1_c00afa428f94 | [2018-12-07T13:44:43,394][INFO ][o.e.n.Node ] [] initializing ... api_1_d0ea38fddddd | goroutine 1 [running]: es_1_c00afa428f94 | [2018-12-07T13:44:43,454][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] using [1] data paths, mounts [[/ (overlay)]], net usable_space [7.2gb], net total_space [29.4gb], spins? [possibly], types [overlay] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] Reading of all Master_info entries succeded es_1_c00afa428f94 | [2018-12-07T13:44:43,454][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] heap size [1.9gb], compressed ordinary object pointers [true] es_1_c00afa428f94 | [2018-12-07T13:44:43,455][INFO ][o.e.n.Node ] node name [Izt4zdP] derived from node ID [Izt4zdPRTW2IHvngLppy2w]; set [node.name] to override sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] Added new Master_info '' to hash table api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4205934a0, 0x108a829, 0x3, 0xc42041d720, 0x14, 0xc42051e630, 0xe06aa0, 0xc4204411e0, ...) es_1_c00afa428f94 | [2018-12-07T13:44:43,455][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] mysqld: ready for connections. api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3 es_1_c00afa428f94 | [2018-12-07T13:44:43,455][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch] sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+maria~jessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequest(0x0, 0x189d380, 0xc4205934a0, 0x108a829, 0x3, 0xc42041d720, 0x14, 0xc42051e630, 0xe06aa0, 0xc4204411e0, ...) es_1_c00afa428f94 | [2018-12-07T13:44:44,605][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [aggs-matrix-stats] sql_1_baad39f9a608 | 2018-12-06 14:43:38 140549600286464 [Note] InnoDB: Buffer pool(s) load completed at 181206 14:43:38 api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f es_1_c00afa428f94 | [2018-12-07T13:44:44,605][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [ingest-common] sql_1_baad39f9a608 | 2018-12-06 14:45:32 140550325155584 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.2' (Got an error reading communication packets) api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc420543e38, 0x189d380, 0xc4205934a0, 0x189d380, 0xc4205934a0, 0xc4204411a0) es_1_c00afa428f94 | [2018-12-07T13:44:44,605][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-expression] sql_1_baad39f9a608 | 2018-12-06 15:30:52 140550325155584 [Note] mysqld (unknown): Normal shutdown api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111 es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-groovy] sql_1_baad39f9a608 | api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1() es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-mustache] sql_1_baad39f9a608 | 2018-12-06 15:30:52 140550325155584 [Note] Event Scheduler: Purging the queue. 0 events api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-painless] sql_1_baad39f9a608 | 2018-12-06 15:30:52 140549642249984 [Note] InnoDB: FTS optimize thread exiting. api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init() es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [parent-join] sql_1_baad39f9a608 | 2018-12-06 15:30:52 140550325155584 [Note] InnoDB: Starting shutdown... api_1_d0ea38fddddd | :1 +0x183 es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [percolator] sql_1_baad39f9a608 | 2018-12-06 15:30:52 140549600286464 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init() es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [reindex] sql_1_baad39f9a608 | 2018-12-06 15:30:52 140549600286464 [Note] InnoDB: Buffer pool(s) dump completed at 181206 15:30:52 api_1_d0ea38fddddd | :1 +0x8c es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty3] sql_1_baad39f9a608 | 2018-12-06 15:30:53 140550325155584 [Note] InnoDB: Shutdown completed; log sequence number 2179632 api_1_d0ea38fddddd | main.init() api_1_d0ea38fddddd | :1 +0x78 sql_1_baad39f9a608 | 2018-12-06 15:30:53 140550325155584 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty4] sql_1_baad39f9a608 | 2018-12-06 15:30:53 140550325155584 [Note] mysqld: Shutdown complete sql_1_baad39f9a608 | es_1_c00afa428f94 | [2018-12-07T13:44:44,607][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-geoip] sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 1 ... es_1_c00afa428f94 | [2018-12-07T13:44:44,607][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-user-agent] sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins es_1_c00afa428f94 | [2018-12-07T13:44:44,607][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [x-pack] sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Uses event mutexes es_1_c00afa428f94 | [2018-12-07T13:44:47,140][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/56] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Compressed tables use zlib 1.2.8 es_1_c00afa428f94 | [2018-12-07T13:44:47,177][INFO ][o.e.d.DiscoveryModule ] [Izt4zdP] using discovery type [zen] sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Using Linux native AIO es_1_c00afa428f94 | [2018-12-07T13:44:47,781][INFO ][o.e.n.Node ] initialized sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Number of pools: 1 es_1_c00afa428f94 | [2018-12-07T13:44:47,781][INFO ][o.e.n.Node ] [Izt4zdP] starting ... sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Using SSE2 crc32 instructions es_1_c00afa428f94 | [2018-12-07T13:44:47,949][INFO ][o.e.t.TransportService ] [Izt4zdP] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300} sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M es_1_c00afa428f94 | [2018-12-07T13:44:47,955][INFO ][o.e.b.BootstrapChecks ] [Izt4zdP] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Completed initialization of buffer pool es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed sql_1_baad39f9a608 | 2018-12-06 15:32:41 139756584908544 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Highest supported file format is Barracuda. sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: 128 out of 128 rollback segments are active. es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Creating shared tablespace for temporary tables sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... es_1_c00afa428f94 | [2018-12-07T13:44:47,962][INFO ][o.e.n.Node ] [Izt4zdP] stopping ... sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: File './ibtmp1' size is now 12 MB. trackit_api_1_d0ea38fddddd exited with code 2 sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: 5.7.19 started; log sequence number 2179632 es_1_c00afa428f94 | [2018-12-07T13:44:48,048][INFO ][o.e.n.Node ] [Izt4zdP] stopped es_1_c00afa428f94 | [2018-12-07T13:44:48,048][INFO ][o.e.n.Node ] [Izt4zdP] closing ... sql_1_baad39f9a608 | 2018-12-06 15:32:41 139756012824320 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool es_1_c00afa428f94 | [2018-12-07T13:44:48,055][INFO ][o.e.n.Node ] [Izt4zdP] closed sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] Plugin 'FEEDBACK' is disabled. es_1_c00afa428f94 | [2018-12-07T13:44:48,057][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] Server socket created on IP: '::'. sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode. sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] Reading of all Master_info entries succeded sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] Added new Master_info '' to hash table sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] mysqld: ready for connections. sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+maria~jessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution sql_1_baad39f9a608 | 2018-12-06 15:32:41 139756012824320 [Note] InnoDB: Buffer pool(s) load completed at 181206 15:32:41 sql_1_baad39f9a608 | 2018-12-06 15:34:30 139757167404800 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets) sql_1_baad39f9a608 | 2018-12-07 13:42:52 139757167404800 [Note] mysqld (unknown): Normal shutdown sql_1_baad39f9a608 | sql_1_baad39f9a608 | 2018-12-07 13:42:52 139757167404800 [Note] Event Scheduler: Purging the queue. 0 events sql_1_baad39f9a608 | 2018-12-07 13:42:52 139756465800960 [Note] InnoDB: FTS optimize thread exiting. sql_1_baad39f9a608 | 2018-12-07 13:42:52 139757167404800 [Note] InnoDB: Starting shutdown... sql_1_baad39f9a608 | 2018-12-07 13:42:52 139756012824320 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool sql_1_baad39f9a608 | 2018-12-07 13:42:52 139756012824320 [Note] InnoDB: Buffer pool(s) dump completed at 181207 13:42:52 sql_1_baad39f9a608 | 2018-12-07 13:42:54 139757167404800 [Note] InnoDB: Shutdown completed; log sequence number 2179660 sql_1_baad39f9a608 | 2018-12-07 13:42:54 139757167404800 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" sql_1_baad39f9a608 | 2018-12-07 13:42:54 139757167404800 [Note] mysqld: Shutdown complete sql_1_baad39f9a608 | sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 1 ... sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Uses event mutexes sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Compressed tables use zlib 1.2.8 sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Using Linux native AIO sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Number of pools: 1 sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Using SSE2 crc32 instructions sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Completed initialization of buffer pool trackit_es_1_c00afa428f94 exited with code 78 sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436050282240 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Highest supported file format is Barracuda. sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: 128 out of 128 rollback segments are active. sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Creating shared tablespace for temporary tables sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: File './ibtmp1' size is now 12 MB. sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: 5.7.19 started; log sequence number 2179660 sql_1_baad39f9a608 | 2018-12-07 13:44:42 140435674613504 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] Plugin 'FEEDBACK' is disabled. sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] Server socket created on IP: '::'. sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode. sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] Reading of all Master_info entries succeded sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] Added new Master_info '' to hash table sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] mysqld: ready for connections. sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+maria~jessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution sql_1_baad39f9a608 | 2018-12-07 13:44:42 140435674613504 [Note] InnoDB: Buffer pool(s) load completed at 181207 13:44:42 sql_1_baad39f9a608 | 2018-12-07 13:46:33 140436136883968 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets) sql_1_baad39f9a608 | 2018-12-07 13:49:07 140436136580864 [Note] mysqld (unknown): Normal shutdown sql_1_baad39f9a608 | sql_1_baad39f9a608 | 2018-12-07 13:49:07 140436136580864 [Note] Event Scheduler: Purging the queue. 0 events sql_1_baad39f9a608 | 2018-12-07 13:49:07 140435926263552 [Note] InnoDB: FTS optimize thread exiting. sql_1_baad39f9a608 | 2018-12-07 13:49:07 140436136580864 [Note] InnoDB: Starting shutdown... sql_1_baad39f9a608 | 2018-12-07 13:49:07 140435674613504 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool sql_1_baad39f9a608 | 2018-12-07 13:49:07 140435674613504 [Note] InnoDB: Buffer pool(s) dump completed at 181207 13:49:07 sql_1_baad39f9a608 | 2018-12-07 13:49:09 140436136580864 [Note] InnoDB: Shutdown completed; log sequence number 2179688 sql_1_baad39f9a608 | 2018-12-07 13:49:09 140436136580864 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" sql_1_baad39f9a608 | 2018-12-07 13:49:09 140436136580864 [Note] mysqld: Shutdown complete sql_1_baad39f9a608 | sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 1 ... sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Uses event mutexes sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Compressed tables use zlib 1.2.8 sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Using Linux native AIO sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Number of pools: 1 sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Using SSE2 crc32 instructions sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Completed initialization of buffer pool sql_1_baad39f9a608 | 2018-12-07 13:49:10 140542484383488 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Highest supported file format is Barracuda. sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: 128 out of 128 rollback segments are active. sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: Creating shared tablespace for temporary tables sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: File './ibtmp1' size is now 12 MB. sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: 5.7.19 started; log sequence number 2179688 sql_1_baad39f9a608 | 2018-12-07 13:49:11 140542243489536 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] Plugin 'FEEDBACK' is disabled. sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] Server socket created on IP: '::'. sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode. sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] Reading of all Master_info entries succeded sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] Added new Master_info '' to hash table sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] mysqld: ready for connections. sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+maria~jessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution sql_1_baad39f9a608 | 2018-12-07 13:49:11 140542243489536 [Note] InnoDB: Buffer pool(s) load completed at 181207 13:49:11

thibautcornolti commented 5 years ago

Your error seems to be: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

According to elastic.co, you should try sudo ulimit -n 65536.

sathyajackson commented 5 years ago

I tired but the still the same. Please find the logs below.

docker-compose logs -f WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string. WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string. WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string. WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string. Attaching to trackit_api_1_d0ea38fddddd, trackit_es_1_c00afa428f94, trackit_sql_1_baad39f9a608, trackit_ui_1_c7e00a815717 api_1_d0ea38fddddd | {"level":"info","time":"2018-12-07T15:33:45.49718687Z","message":"Successfully connected to SQL database."} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:33:50.58063346Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:33:57.9534805Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:05.594416443Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:13.105674314Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-07T15:33:46,062][INFO ][o.e.n.Node ] [] initializing ... api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:20.756037608Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-07T15:33:46,128][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] using [1] data paths, mounts [[/ (overlay)]], net usable_space [5.3gb], net total_space [29.4gb], spins? [possibly], types [overlay] sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 1 ... api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:28.334634362Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-07T15:33:46,128][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] heap size [1.9gb], compressed ordinary object pointers [true] sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:35.813448415Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-07T15:33:46,130][INFO ][o.e.n.Node ] node name [W3u251Y] derived from node ID [W3u251YpTuK25GAuM7O-ng]; set [node.name] to override ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security. sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Uses event mutexes es_1_c00afa428f94 | [2018-12-07T15:33:46,130][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:43.004810852Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching ' sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Compressed tables use zlib 1.2.8 es_1_c00afa428f94 | [2018-12-07T15:33:46,130][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:50.309950866Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} ui_1_c7e00a815717 | 2018-12-07 15:33:44,258 CRIT Supervisor running as root (no user in config file) sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Using Linux native AIO es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [aggs-matrix-stats] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:57.562354239Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} ui_1_c7e00a815717 | 2018-12-07 15:33:44,260 INFO supervisord started with pid 1 sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Number of pools: 1 es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [ingest-common] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:04.956597664Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} ui_1_c7e00a815717 | 2018-12-07 15:33:45,262 INFO spawned: 'webui' with pid 9 sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Using SSE2 crc32 instructions es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-expression] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:12.103780063Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} ui_1_c7e00a815717 | 2018-12-07 15:33:46,470 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-groovy] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:19.262813137Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Completed initialization of buffer pool api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:26.72686607Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-mustache] sql_1_baad39f9a608 | 2018-12-07 15:33:44 139770535020288 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:34.252869193Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-painless] sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Highest supported file format is Barracuda. api_1_d0ea38fddddd | {"level":"error","time":"2018-12-07T15:35:36.25299783Z","message":"Failed to connect to ElasticSearch database. Not retrying."} es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [parent-join] sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: 128 out of 128 rollback segments are active. api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [percolator] sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: Creating shared tablespace for temporary tables api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33] es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [reindex] sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... api_1_d0ea38fddddd | sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: File './ibtmp1' size is now 12 MB. es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty3] api_1_d0ea38fddddd | goroutine 1 [running]: sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: 5.7.19 started; log sequence number 2179744 es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty4] api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4204feea0, 0x108a829, 0x3, 0xc42046de40, 0x14, 0xc420554b70, 0xe06aa0, 0xc420323cc0, ...) sql_1_baad39f9a608 | 2018-12-07 15:33:45 139770156009216 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool es_1_c00afa428f94 | [2018-12-07T15:33:47,294][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-geoip] api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3 sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] Plugin 'FEEDBACK' is disabled. api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequest(0x0, 0x189d380, 0xc4204feea0, 0x108a829, 0x3, 0xc42046de40, 0x14, 0xc420554b70, 0xe06aa0, 0xc420323cc0, ...) sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] Server socket created on IP: '::'. es_1_c00afa428f94 | [2018-12-07T15:33:47,294][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-user-agent] api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode. es_1_c00afa428f94 | [2018-12-07T15:33:47,294][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [x-pack] api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc420599e38, 0x189d380, 0xc4204feea0, 0x189d380, 0xc4204feea0, 0xc420323c80) sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] Reading of all Master_info entries succeded es_1_c00afa428f94 | [2018-12-07T15:33:49,772][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/57] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111 sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] Added new Master_info '' to hash table es_1_c00afa428f94 | [2018-12-07T15:33:49,799][INFO ][o.e.d.DiscoveryModule ] [W3u251Y] using discovery type [zen] api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1() sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] mysqld: ready for connections. es_1_c00afa428f94 | [2018-12-07T15:33:50,431][INFO ][o.e.n.Node ] initialized api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+maria~jessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution es_1_c00afa428f94 | [2018-12-07T15:33:50,431][INFO ][o.e.n.Node ] [W3u251Y] starting ... api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init() sql_1_baad39f9a608 | 2018-12-07 15:33:45 139770156009216 [Note] InnoDB: Buffer pool(s) load completed at 181207 15:33:45 es_1_c00afa428f94 | [2018-12-07T15:33:50,627][INFO ][o.e.t.TransportService ] [W3u251Y] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300} api_1_d0ea38fddddd | :1 +0x183 sql_1_baad39f9a608 | 2018-12-07 15:35:36 139770621622016 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets) es_1_c00afa428f94 | [2018-12-07T15:33:50,635][INFO ][o.e.b.BootstrapChecks ] [W3u251Y] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init() sql_1_baad39f9a608 | 2018-12-10 9:02:39 139770621622016 [Warning] Aborted connection 9 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets) es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed api_1_d0ea38fddddd | :1 +0x8c es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] api_1_d0ea38fddddd | main.init() es_1_c00afa428f94 | [2018-12-07T15:33:50,643][INFO ][o.e.n.Node ] [W3u251Y] stopping ... api_1_d0ea38fddddd | :1 +0x78 es_1_c00afa428f94 | [2018-12-07T15:33:50,736][INFO ][o.e.n.Node ] [W3u251Y] stopped api_1_d0ea38fddddd | {"level":"info","time":"2018-12-10T09:00:48.819069531Z","message":"Successfully connected to SQL database."} es_1_c00afa428f94 | [2018-12-07T15:33:50,736][INFO ][o.e.n.Node ] [W3u251Y] closing ... api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:00:53.87354117Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-07T15:33:50,745][INFO ][o.e.n.Node ] [W3u251Y] closed api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:01.282592249Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-07T15:33:50,746][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:08.929072086Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:50,591][INFO ][o.e.n.Node ] [] initializing ... api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:16.365412968Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:23.89397026Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:50,651][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] using [1] data paths, mounts [[/ (overlay)]], net usable_space [5gb], net total_space [29.4gb], spins? [possibly], types [overlay] es_1_c00afa428f94 | [2018-12-10T09:00:50,651][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] heap size [1.9gb], compressed ordinary object pointers [true] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:31.411812725Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:50,652][INFO ][o.e.n.Node ] node name [W3u251Y] derived from node ID [W3u251YpTuK25GAuM7O-ng]; set [node.name] to override es_1_c00afa428f94 | [2018-12-10T09:00:50,652][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:38.941477556Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:50,652][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:46.316465352Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [aggs-matrix-stats] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:53.781694708Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [ingest-common] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:01.259192326Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-expression] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:08.70139361Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-groovy] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:16.005113075Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-mustache] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:23.234197869Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-painless] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:30.39815982Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [parent-join] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:37.897757971Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [percolator] api_1_d0ea38fddddd | {"level":"error","time":"2018-12-10T09:02:39.897985891Z","message":"Failed to connect to ElasticSearch database. Not retrying."} es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [reindex] api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty3] api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33] es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty4] api_1_d0ea38fddddd | es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-geoip] es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-user-agent] api_1_d0ea38fddddd | goroutine 1 [running]: es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [x-pack] api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc42055b800, 0x108a829, 0x3, 0xc4202fc440, 0x14, 0xc4204dc3c0, 0xe06aa0, 0xc42046c160, ...) es_1_c00afa428f94 | [2018-12-10T09:00:54,314][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3 api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequest(0x0, 0x189d380, 0xc42055b800, 0x108a829, 0x3, 0xc4202fc440, 0x14, 0xc4204dc3c0, 0xe06aa0, 0xc42046c160, ...) es_1_c00afa428f94 | [2018-12-10T09:00:54,348][INFO ][o.e.d.DiscoveryModule ] [W3u251Y] using discovery type [zen] es_1_c00afa428f94 | [2018-12-10T09:00:54,969][INFO ][o.e.n.Node ] initialized api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f es_1_c00afa428f94 | [2018-12-10T09:00:54,969][INFO ][o.e.n.Node ] [W3u251Y] starting ... api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc420599e38, 0x189d380, 0xc42055b800, 0x189d380, 0xc42055b800, 0xc42046c100) es_1_c00afa428f94 | [2018-12-10T09:00:55,164][INFO ][o.e.t.TransportService ] [W3u251Y] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300} api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111 es_1_c00afa428f94 | [2018-12-10T09:00:55,170][INFO ][o.e.b.BootstrapChecks ] [W3u251Y] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1() es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init() es_1_c00afa428f94 | [2018-12-10T09:00:55,177][INFO ][o.e.n.Node ] [W3u251Y] stopping ... api_1_d0ea38fddddd | :1 +0x183 es_1_c00afa428f94 | [2018-12-10T09:00:55,253][INFO ][o.e.n.Node ] [W3u251Y] stopped api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init() es_1_c00afa428f94 | [2018-12-10T09:00:55,253][INFO ][o.e.n.Node ] [W3u251Y] closing ... api_1_d0ea38fddddd | :1 +0x8c es_1_c00afa428f94 | [2018-12-10T09:00:55,260][INFO ][o.e.n.Node ] [W3u251Y] closed api_1_d0ea38fddddd | main.init() api_1_d0ea38fddddd | :1 +0x78 es_1_c00afa428f94 | [2018-12-10T09:00:55,262][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started api_1_d0ea38fddddd | {"level":"info","time":"2018-12-10T09:03:19.337682382Z","message":"Successfully connected to SQL database."} es_1_c00afa428f94 | [2018-12-10T09:03:21,082][INFO ][o.e.n.Node ] [] initializing ... api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:24.393296692Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:03:21,135][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] using [1] data paths, mounts [[/ (overlay)]], net usable_space [4.6gb], net total_space [29.4gb], spins? [possibly], types [overlay] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:31.756960454Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:03:21,135][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] heap size [1.9gb], compressed ordinary object pointers [true] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:39.032258799Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:03:21,136][INFO ][o.e.n.Node ] node name [W3u251Y] derived from node ID [W3u251YpTuK25GAuM7O-ng]; set [node.name] to override api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:46.260197022Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:53.553834862Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:03:21,136][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16] api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:04:00.791731735Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} es_1_c00afa428f94 | [2018-12-10T09:03:21,136][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [aggs-matrix-stats] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [ingest-common] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-expression] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-groovy] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-mustache] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-painless] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [parent-join] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [percolator] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [reindex] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty3] es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty4] es_1_c00afa428f94 | [2018-12-10T09:03:22,157][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-geoip] es_1_c00afa428f94 | [2018-12-10T09:03:22,157][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-user-agent] es_1_c00afa428f94 | [2018-12-10T09:03:22,157][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [x-pack] es_1_c00afa428f94 | [2018-12-10T09:03:24,671][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/56] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV es_1_c00afa428f94 | [2018-12-10T09:03:24,700][INFO ][o.e.d.DiscoveryModule ] [W3u251Y] using discovery type [zen] es_1_c00afa428f94 | [2018-12-10T09:03:25,200][INFO ][o.e.n.Node ] initialized es_1_c00afa428f94 | [2018-12-10T09:03:25,200][INFO ][o.e.n.Node ] [W3u251Y] starting ... es_1_c00afa428f94 | [2018-12-10T09:03:25,349][INFO ][o.e.t.TransportService ] [W3u251Y] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300} es_1_c00afa428f94 | [2018-12-10T09:03:25,355][INFO ][o.e.b.BootstrapChecks ] [W3u251Y] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] es_1_c00afa428f94 | [2018-12-10T09:03:25,363][INFO ][o.e.n.Node ] [W3u251Y] stopping ... es_1_c00afa428f94 | [2018-12-10T09:03:25,463][INFO ][o.e.n.Node ] [W3u251Y] stopped es_1_c00afa428f94 | [2018-12-10T09:03:25,463][INFO ][o.e.n.Node ] [W3u251Y] closing ... es_1_c00afa428f94 | [2018-12-10T09:03:25,470][INFO ][o.e.n.Node ] [W3u251Y] closed es_1_c00afa428f94 | [2018-12-10T09:03:25,472][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started trackit_es_1_c00afa428f94 exited with code 78 api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:04:08.23391975Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:04:15.423108323Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}

I am using amazon linux to test this.

sathyajackson commented 5 years ago

api_1_d0ea38fddddd | goroutine 1 [running]: api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4205a0e40, 0x108a829, 0x3, 0xc420528ac0, 0x14, 0xc42059def0, 0xe06aa0, 0xc420437730, ...) api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3 api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(Client).PerformRequest(0x0, 0x189d380, 0xc4205a0e40, 0x108a829, 0x3, 0xc420528ac0, 0x14, 0xc42059def0, 0xe06aa0, 0xc420437730, ...) api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc42054be38, 0x189d380, 0xc4205a0e40, 0x189d380, 0xc4205a0e40, 0xc4204376f0) api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111 api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1() api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init() api_1_d0ea38fddddd | :1 +0x183 api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init() api_1_d0ea38fddddd | :1 +0x8c api_1_d0ea38fddddd | main.init() api_1_d0ea38fddddd | :1 +0x78 sql_1_baad39f9a608 | 2018-12-10 9:05:08 139770621622016 [Warning] Aborted connection 10 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets)

sathyajackson commented 5 years ago

docker ps output

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ef3d4e53cd3b trackit_sql "docker-entrypoint.s…" 2 days ago Up 2 days 127.0.0.1:3306->3306/tcp trackit_sql_1_baad39f9a608 8aaf42dd85ce trackit_ui "/usr/bin/supervisord" 2 days ago Up 2 days 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp trackit_ui_1_c7e00a815717

sathyajackson commented 5 years ago

I did make some modification on the docker-compose file to point the API to server public IP. Still i see the below error and my ealstic container is not running.

api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:23.123310772Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:30.404844304Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:37.880940548Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:45.299431229Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:52.662917352Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:00.176747145Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:07.420307808Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:14.782587565Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:22.05947155Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:29.313760388Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:36.713434361Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:43.874558415Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:51.17893862Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:58.227660307Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}

sathyajackson commented 5 years ago

Not sure if this is related to the problem. but here is the output i get when i try everything fresh.

es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] es_1_c00afa428f94 | [2018-12-10T09:40:57,584][INFO ][o.e.n.Node ] [W3u251Y] stopping ... es_1_c00afa428f94 | [2018-12-10T09:40:57,684][INFO ][o.e.n.Node ] [W3u251Y] stopped es_1_c00afa428f94 | [2018-12-10T09:40:57,684][INFO ][o.e.n.Node ] [W3u251Y] closing ... es_1_c00afa428f94 | [2018-12-10T09:40:57,692][INFO ][o.e.n.Node ] [W3u251Y] closed es_1_c00afa428f94 | [2018-12-10T09:40:57,694][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started trackit_es_1_c00afa428f94 exited with code 78 api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:42:16.356009835Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:42:23.416671448Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"} api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:42:30.583070176Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}

sathyajackson commented 5 years ago

i tried to install the same in a fresh machine and i am now getting the below python execption.

===> pulling the docker containers Traceback (most recent call last): File "./trackit-server/scripts/awsenv", line 31, in main(profile, cmd) File "./trackit-server/scripts/awsenv", line 16, in main profile_config = get_profile_config(profile) File "./trackit-server/scripts/awsenv", line 13, in get_profile_config return c[profile] File "/usr/lib64/python3.6/configparser.py", line 959, in getitem raise KeyError(key) KeyError: 'default' ===> building containers Traceback (most recent call last): File "./trackit-server/scripts/awsenv", line 31, in main(profile, cmd) File "./trackit-server/scripts/awsenv", line 16, in main profile_config = get_profile_config(profile) File "./trackit-server/scripts/awsenv", line 13, in get_profile_config return c[profile] File "/usr/lib64/python3.6/configparser.py", line 959, in getitem raise KeyError(key) KeyError: 'default' ===> starting trackit Traceback (most recent call last): File "./trackit-server/scripts/awsenv", line 31, in main(profile, cmd) File "./trackit-server/scripts/awsenv", line 16, in main profile_config = get_profile_config(profile) File "./trackit-server/scripts/awsenv", line 13, in get_profile_config return c[profile] File "/usr/lib64/python3.6/configparser.py", line 959, in getitem raise KeyError(key) KeyError: 'default'

thibautcornolti commented 5 years ago

You need to configure aws-cli: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html Then, try again sudo ulimit -n 65536 and sudo sysctl -w fs.file-max=65536

Please write your logs in the appropriate format so we can read it.

image will be shown as:

[my logs]
sathyajackson commented 5 years ago

Hi thanks for the input. I have already configured the Access_key and secreat_key using aws cli on the same server where i am running trackit.

Still when i start the docker containers i am getting the below message

ARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string. WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string. WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string. WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.

thibautcornolti commented 5 years ago

Hi, are you sure you start TrackIt with our start.sh script?

sathyajackson commented 5 years ago

Yes I do with start.sh

thibautcornolti commented 5 years ago

Look in the file ~/.aws/credentials, do you have a default profile?

sathyajackson commented 5 years ago

Yes, I do have that and I even check the credentials and I works fine with aws-cli

thibautcornolti commented 5 years ago

./trackit-server/scripts/awsenv default env | grep 'AWS_' | cut -f1 -d"="

Does it display this?

AWS_DEFAULT_REGION
AWS_ACCESS_KEY_ID
AWS_SESSION_TOKEN
AWS_SECRET_ACCESS_KEY
sathyajackson commented 5 years ago

I get the following error

Traceback (most recent call last): File "./awsenv", line 31, in main(profile, cmd) File "./awsenv", line 16, in main profile_config = get_profile_config(profile) File "./awsenv", line 13, in get_profile_config return c[profile] File "/usr/lib64/python3.6/configparser.py", line 959, in getitem raise KeyError(key) KeyError: 'default'

sathyajackson commented 5 years ago

sorry for the above statement.

Now i got to see what is expected.

AWS_CLOUDWATCH_HOME AWS_PATH AWS_AUTO_SCALING_HOME AWS_ELB_HOME AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

thibautcornolti commented 5 years ago

Ok!

So let's start again, what are the start.sh and the containers outputs? Please write your logs as I wrote here https://github.com/trackit/trackit/issues/69#issuecomment-449040526

Thanks

sathyajackson commented 5 years ago

Alrite here you go.

./start.sh
===> checking vm.max_map_count setting
===> updating git submodules
Adding existing repo at 'trackit2-client' to the index
Adding existing repo at 'trackit-server' to the index
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), done.
From https://github.com/trackit/trackit-server
   3b379b6..e1b2aca  master     -> origin/master
Submodule path 'trackit-server': checked out 'e1b2acafc2d678932d0b97d46aa3c6debe9b4258'
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), done.
From https://github.com/trackit/trackit2-client
   5f433fb..a3b72bc  master     -> origin/master
Submodule path 'trackit2-client': checked out 'a3b72bc0a1a83824a2ce923b4907618fb16ac068'
/data/trackit/scripts /data/trackit
===> building intermediate container for server compilation
Sending build context to Docker daemon  9.185MB
Step 1/13 : FROM ubuntu:16.04
 ---> a51debf7e1eb
Step 2/13 : MAINTAINER Romain Zanchi <romain@trackit.io>
 ---> Using cache
 ---> e63c6835f86f
Step 3/13 : RUN apt-get update
 ---> Using cache
 ---> aec3f4a32c4f
Step 4/13 : RUN apt-get install -y curl git
 ---> Using cache
 ---> c1147dd473cb
Step 5/13 : WORKDIR /
 ---> Using cache
 ---> 3df8f0292676
Step 6/13 : RUN curl -O https://dl.google.com/go/go1.9.2.linux-amd64.tar.gz
 ---> Using cache
 ---> 93a76e81b7f9
Step 7/13 : RUN tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz
 ---> Using cache
 ---> ae722739f0e1
Step 8/13 : ENV PATH="/usr/local/go/bin:${PATH}"
 ---> Using cache
 ---> 13ca92816530
Step 9/13 : RUN go get -u github.com/kardianos/govendor
 ---> Using cache
 ---> 44a831cbcbcb
Step 10/13 : ENV PATH="/root/go/bin:${PATH}"
 ---> Using cache
 ---> 7ab7a1ad79a1
Step 11/13 : COPY . /root/go/src/github.com/trackit
 ---> 4e45d507d712
Step 12/13 : COPY compiler/compile_server.sh /compile_server.sh
 ---> 96420754d1be
Step 13/13 : ENTRYPOINT ["/compile_server.sh"]
 ---> Running in 639b029ff2f9
Removing intermediate container 639b029ff2f9
 ---> 8ac7cd80b58b
Successfully built 8ac7cd80b58b
Successfully tagged server_compiler:latest
===> fetching dependencies
fetch "github.com/alexflint/go-arg"
fetch "github.com/alexflint/go-scalar"
fetch "github.com/aws/aws-sdk-go/aws"
fetch "github.com/aws/aws-sdk-go/aws/awserr"
fetch "github.com/aws/aws-sdk-go/aws/awsutil"
fetch "github.com/aws/aws-sdk-go/aws/client"
fetch "github.com/aws/aws-sdk-go/aws/client/metadata"
fetch "github.com/aws/aws-sdk-go/aws/corehandlers"
fetch "github.com/aws/aws-sdk-go/aws/credentials"
fetch "github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
fetch "github.com/aws/aws-sdk-go/aws/credentials/endpointcreds"
fetch "github.com/aws/aws-sdk-go/aws/credentials/stscreds"
fetch "github.com/aws/aws-sdk-go/aws/csm"
fetch "github.com/aws/aws-sdk-go/aws/defaults"
fetch "github.com/aws/aws-sdk-go/aws/ec2metadata"
fetch "github.com/aws/aws-sdk-go/aws/endpoints"
fetch "github.com/aws/aws-sdk-go/aws/request"
fetch "github.com/aws/aws-sdk-go/aws/session"
fetch "github.com/aws/aws-sdk-go/aws/signer/v4"
fetch "github.com/aws/aws-sdk-go/internal/sdkio"
fetch "github.com/aws/aws-sdk-go/internal/sdkrand"
fetch "github.com/aws/aws-sdk-go/internal/sdkuri"
fetch "github.com/aws/aws-sdk-go/internal/shareddefaults"
fetch "github.com/aws/aws-sdk-go/private/protocol"
fetch "github.com/aws/aws-sdk-go/private/protocol/ec2query"
fetch "github.com/aws/aws-sdk-go/private/protocol/eventstream"
fetch "github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi"
fetch "github.com/aws/aws-sdk-go/private/protocol/json/jsonutil"
fetch "github.com/aws/aws-sdk-go/private/protocol/json/jsonutil"
fetch "github.com/aws/aws-sdk-go/private/protocol/jsonrpc"
fetch "github.com/aws/aws-sdk-go/private/protocol/jsonrpc"
fetch "github.com/aws/aws-sdk-go/private/protocol/query"
fetch "github.com/aws/aws-sdk-go/private/protocol/query/queryutil"
fetch "github.com/aws/aws-sdk-go/private/protocol/rest"
fetch "github.com/aws/aws-sdk-go/private/protocol/restjson"
fetch "github.com/aws/aws-sdk-go/private/protocol/restxml"
fetch "github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil"
fetch "github.com/aws/aws-sdk-go/service/cloudwatch"
fetch "github.com/aws/aws-sdk-go/service/costexplorer"
fetch "github.com/aws/aws-sdk-go/service/ec2"
fetch "github.com/aws/aws-sdk-go/service/elasticache"
fetch "github.com/aws/aws-sdk-go/service/elasticsearchservice"
fetch "github.com/aws/aws-sdk-go/service/lambda"
fetch "github.com/aws/aws-sdk-go/service/marketplaceentitlementservice"
fetch "github.com/aws/aws-sdk-go/service/marketplacemetering"
fetch "github.com/aws/aws-sdk-go/service/marketplacemetering"
fetch "github.com/aws/aws-sdk-go/service/organizations"
fetch "github.com/aws/aws-sdk-go/service/rds"
fetch "github.com/aws/aws-sdk-go/service/s3"
fetch "github.com/aws/aws-sdk-go/service/s3/s3iface"
fetch "github.com/aws/aws-sdk-go/service/s3/s3manager"
fetch "github.com/aws/aws-sdk-go/service/sts"
fetch "github.com/denisenkom/go-mssqldb"
fetch "github.com/dgrijalva/jwt-go"
fetch "github.com/gedex/inflector"
fetch "github.com/go-ini/ini"
fetch "github.com/go-sql-driver/mysql"
fetch "github.com/jmespath/go-jmespath"
fetch "github.com/knq/dburl"
fetch "github.com/knq/snaker"
fetch "github.com/knq/xo"
fetch "github.com/knq/xo/internal"
fetch "github.com/knq/xo/loaders"
fetch "github.com/knq/xo/models"
fetch "github.com/knq/xo/tplbin"
fetch "github.com/knq/xoutil"
fetch "github.com/lib/pq"
fetch "github.com/lib/pq/oid"
fetch "github.com/mailru/easyjson"
fetch "github.com/mailru/easyjson/buffer"
fetch "github.com/mailru/easyjson/jlexer"
fetch "github.com/mailru/easyjson/jwriter"
fetch "github.com/mattn/go-sqlite3"
fetch "github.com/pkg/errors"
fetch "github.com/satori/go.uuid"
fetch "github.com/sha1sum/aws_signing_client"
fetch "github.com/tealeg/xlsx"
fetch "github.com/trackit/jsonlog"
fetch "golang.org/x/crypto/bcrypt"
fetch "golang.org/x/crypto/blowfish"
fetch "golang.org/x/crypto/md4"
fetch "golang.org/x/net/context"
fetch "golang.org/x/sync/errgroup"
fetch "golang.org/x/tools/cmd/goimports"
fetch "golang.org/x/tools/go/ast/astutil"
fetch "golang.org/x/tools/imports"
fetch "gopkg.in/olivere/elastic.v5"
fetch "gopkg.in/olivere/elastic.v5/config"
fetch "gopkg.in/olivere/elastic.v5/uritemplates"
fetch "gopkg.in/rana/ora.v4"
fetch "gopkg.in/rana/ora.v4/date"
fetch "gopkg.in/rana/ora.v4/num"
===> compiling trackit server
+ rm -f ./server/main
+ [[ ! -f server/main ]]
+ pushd server
+ ./buildstatic.sh
~/go/src/github.com/trackit/trackit-server/server ~/go/src/github.com/trackit/trackit-server
++ git rev-parse --short=8 HEAD
+ build_number=e1b2acaf
++ git status --porcelain
++ wc -l
+ [[ 0 -gt 0 ]]
+ CGO_ENABLED=0
+ GOOS=linux
+ go build -a -installsuffix cgo -ldflags '-X main.buildNumber=e1b2acaf' -o main
~/go/src/github.com/trackit/trackit-server
+ popd
+ mv ./server/main /trackit-server
/data/trackit
/data/trackit/trackit-server/scripts /data/trackit
==> copying schema.sql
+ cp ../db/schema.sql ../docker/sql/schema.sql
/data/trackit
===> pulling the docker containers
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Pulling ui  ... done
Pulling es  ... done
Pulling sql ... done
Pulling api ... done
===> building containers
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
es uses an image, skipping
Building ui
Step 1/14 : FROM node:6-slim
 ---> 39cfb4866972
Step 2/14 : ARG API_URL
 ---> Using cache
 ---> d0881fb735be
Step 3/14 : ENV API_URL=${API_URL}
 ---> Using cache
 ---> 1d4053b7669d
Step 4/14 : RUN curl -sS http://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
 ---> Using cache
 ---> c35e9d70f1d9
Step 5/14 : RUN echo "deb http://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
 ---> Using cache
 ---> 00d09b07f70f
Step 6/14 : RUN apt-get update && apt-get install -y supervisor git
 ---> Using cache
 ---> c678367493e1
Step 7/14 : RUN mkdir -p /var/log/supervisor
 ---> Using cache
 ---> ad72386c56bf
Step 8/14 : COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
 ---> Using cache
 ---> 39c9ea2aa904
Step 9/14 : WORKDIR /webui
 ---> Using cache
 ---> 228bfd8f714f
Step 10/14 : COPY ./ ./
 ---> 8e461f2d3a14
Step 11/14 : RUN yarn install
 ---> Running in be333f95dd69
yarn install v1.12.3
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents@1.1.2: The platform "linux" is incompatible with this module.
info "fsevents@1.1.2" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 167.28s.
Removing intermediate container be333f95dd69
 ---> 465aeb9be3de
Step 12/14 : RUN yarn run build
 ---> Running in 517da548a7af
yarn run v1.12.3
$ node scripts/build.js
Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  529.02 KB  build/static/js/main.b4beb5e8.js
  47.02 KB   build/static/css/main.7a79ac8c.css

The bundle size is significantly larger than recommended.
Consider reducing it with code splitting: https://goo.gl/9VhYWB
You can also analyze the project dependencies: https://goo.gl/LeUzfb

The project was built assuming it is hosted at /.
You can control this with the homepage field in your package.json.

The build folder is ready to be deployed.
You may serve it with a static server:

  yarn global add serve
  serve -s build

Done in 72.37s.
Removing intermediate container 517da548a7af
 ---> 778e9253d0f1
Step 13/14 : EXPOSE 80 443
 ---> Running in b1888e79231c
Removing intermediate container b1888e79231c
 ---> 516c39c06f46
Step 14/14 : CMD ["/usr/bin/supervisord"]
 ---> Running in e0b192e55350
Removing intermediate container e0b192e55350
 ---> 96ccf2bb4036
Successfully built 96ccf2bb4036
Successfully tagged trackit_ui:latest
Building sql
Step 1/3 : FROM mariadb:10.2.9
 ---> 629d7f88b09c
Step 2/3 : MAINTAINER Victor Schubert <victor@trackit.io>
 ---> Using cache
 ---> 4884858e75c0
Step 3/3 : COPY schema.sql /docker-entrypoint-initdb.d/schema.sql
 ---> f6a1e98c1f2d
Successfully built f6a1e98c1f2d
Successfully tagged trackit_sql:latest
Building api
Step 1/6 : FROM scratch
 --->
Step 2/6 : MAINTAINER Victor Schubert <victor@trackit.io>
 ---> Using cache
 ---> bcfaae81a786
Step 3/6 : EXPOSE 80
 ---> Using cache
 ---> 2a1d3558c4b5
Step 4/6 : COPY ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
 ---> Using cache
 ---> f38cf1c6e20b
Step 5/6 : COPY main /main
 ---> 60d2d487a1d1
Step 6/6 : ENTRYPOINT ["/main"]
 ---> Running in 07a47a834966
Removing intermediate container 07a47a834966
 ---> fe4a1bbd59f2
Successfully built fe4a1bbd59f2
Successfully tagged trackit_api:latest
===> starting trackit
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Creating network "trackit_app" with driver "bridge"
Creating trackit_sql_1 ... done
Creating trackit_ui_1  ... done
Creating trackit_es_1  ... done
Creating trackit_api_1 ... done
===> installation completed, TrackIt is now running on port 80
thibautcornolti commented 5 years ago

Ok! Everything seems to have started well.

Could you send me the containers logs now? Get them by typing docker-compose logs -f Still can't login?

sathyajackson commented 5 years ago

Yes still cant login here are the logs of docker-compose.

WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Attaching to trackit_api_1, trackit_ui_1, trackit_es_1, trackit_sql_1
sql_1  | Initializing database
sql_1  |
sql_1  | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
sql_1  | To do so, start the server, then issue the following commands:
api_1  | {"level":"warning","time":"2019-01-03T12:39:03.303926336Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
api_1  | {"level":"warning","time":"2019-01-03T12:39:05.305095525Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
api_1  | {"level":"warning","time":"2019-01-03T12:39:07.311573427Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
api_1  | {"level":"warning","time":"2019-01-03T12:39:09.312323275Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
sql_1  |
es_1   | [2019-01-03T12:39:05,384][INFO ][o.e.n.Node               ] [] initializing ...
ui_1   | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
ui_1   |   'Supervisord is running as root and it is searching '
sql_1  | '/usr/bin/mysqladmin' -u root password 'new-password'
es_1   | [2019-01-03T12:39:05,464][INFO ][o.e.e.NodeEnvironment    ] [SDWVkXe] using [1] data paths, mounts [[/ (overlay)]], net usable_space [5.7gb], net total_space [29.4gb], spins? [possibly], types [overlay]
ui_1   | 2019-01-03 12:39:02,601 CRIT Supervisor running as root (no user in config file)
api_1  | {"level":"warning","time":"2019-01-03T12:39:11.313068565Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
sql_1  | '/usr/bin/mysqladmin' -u root -h  password 'new-password'
sql_1  |
ui_1   | 2019-01-03 12:39:02,640 INFO supervisord started with pid 1
api_1  | {"level":"warning","time":"2019-01-03T12:39:13.313837089Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
sql_1  | Alternatively you can run:
es_1   | [2019-01-03T12:39:05,464][INFO ][o.e.e.NodeEnvironment    ] [SDWVkXe] heap size [1.9gb], compressed ordinary object pointers [true]
ui_1   | 2019-01-03 12:39:03,642 INFO spawned: 'webui' with pid 8
ui_1   | 2019-01-03 12:39:05,481 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
sql_1  | '/usr/bin/mysql_secure_installation'
es_1   | [2019-01-03T12:39:05,465][INFO ][o.e.n.Node               ] node name [SDWVkXe] derived from node ID [SDWVkXeIS7O3bPvhaSB7jA]; set [node.name] to override
api_1  | {"level":"info","time":"2019-01-03T12:39:15.31513217Z","message":"Successfully connected to SQL database."}
sql_1  |
es_1   | [2019-01-03T12:39:05,466][INFO ][o.e.n.Node               ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.77-70.59.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
api_1  | {"level":"warning","time":"2019-01-03T12:39:20.939298355Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | which will also give you the option of removing the test
es_1   | [2019-01-03T12:39:05,466][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
api_1  | {"level":"warning","time":"2019-01-03T12:39:28.44700456Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | databases and anonymous user created by default.  This is
es_1   | [2019-01-03T12:39:10,562][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [aggs-matrix-stats]
api_1  | {"level":"warning","time":"2019-01-03T12:39:36.078019018Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | strongly recommended for production servers.
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [ingest-common]
api_1  | {"level":"warning","time":"2019-01-03T12:39:43.636509336Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  |
sql_1  | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
api_1  | {"level":"warning","time":"2019-01-03T12:39:51.009755884Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [lang-expression]
sql_1  | MySQL manual for more instructions.
api_1  | {"level":"warning","time":"2019-01-03T12:39:58.613938228Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [lang-groovy]
sql_1  |
api_1  | {"level":"warning","time":"2019-01-03T12:40:06.253430883Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [lang-mustache]
sql_1  | Please report any problems at http://mariadb.org/jira
api_1  | {"level":"warning","time":"2019-01-03T12:40:13.941900517Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [lang-painless]
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [parent-join]
api_1  | {"level":"warning","time":"2019-01-03T12:40:21.376197802Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  |
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [percolator]
api_1  | {"level":"warning","time":"2019-01-03T12:40:28.759457554Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | The latest information about MariaDB is available at http://mariadb.org/.
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [reindex]
api_1  | {"level":"warning","time":"2019-01-03T12:40:36.017083133Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | You can find additional information about the MySQL part at:
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [transport-netty3]
api_1  | {"level":"warning","time":"2019-01-03T12:40:43.444806839Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | http://dev.mysql.com
es_1   | [2019-01-03T12:39:10,564][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [transport-netty4]
api_1  | {"level":"warning","time":"2019-01-03T12:40:50.874534064Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | Consider joining MariaDB's strong and vibrant community:
es_1   | [2019-01-03T12:39:10,564][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded plugin [ingest-geoip]
api_1  | {"level":"warning","time":"2019-01-03T12:40:58.342928551Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | https://mariadb.org/get-involved/
es_1   | [2019-01-03T12:39:10,564][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded plugin [ingest-user-agent]
es_1   | [2019-01-03T12:39:10,564][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded plugin [x-pack]
api_1  | {"level":"warning","time":"2019-01-03T12:41:05.818648543Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  |
api_1  | {"level":"error","time":"2019-01-03T12:41:07.818852452Z","message":"Failed to connect to ElasticSearch database. Not retrying."}
sql_1  | Database initialized
es_1   | [2019-01-03T12:39:14,244][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/48] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
api_1  | panic: runtime error: invalid memory address or nil pointer dereference
sql_1  | MySQL init process in progress...
es_1   | [2019-01-03T12:39:14,291][INFO ][o.e.d.DiscoveryModule    ] [SDWVkXe] using discovery type [zen]
api_1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a4d33]
api_1  |
es_1   | [2019-01-03T12:39:15,160][INFO ][o.e.n.Node               ] initialized
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 97 ...
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
es_1   | [2019-01-03T12:39:15,160][INFO ][o.e.n.Node               ] [SDWVkXe] starting ...
api_1  | goroutine 1 [running]:
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Uses event mutexes
es_1   | [2019-01-03T12:39:15,446][INFO ][o.e.t.TransportService   ] [SDWVkXe] publish_address {10.19.84.3:9300}, bound_addresses {0.0.0.0:9300}
api_1  | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequestWithOptions(0x0, 0x1a57380, 0xc42051a720, 0x119070c, 0x3, 0xc4203a6a40, 0x14, 0xc42026a0c0, 0xecebc0, 0xc4202cf1b0, ...)
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Compressed tables use zlib 1.2.8
es_1   | [2019-01-03T12:39:15,455][INFO ][o.e.b.BootstrapChecks    ] [SDWVkXe] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
api_1  |        /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Using Linux native AIO
es_1   | ERROR: [1] bootstrap checks failed
api_1  | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequest(0x0, 0x1a57380, 0xc42051a720, 0x119070c, 0x3, 0xc4203a6a40, 0x14, 0xc42026a0c0, 0xecebc0, 0xc4202cf1b0, ...)
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Number of pools: 1
es_1   | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
api_1  |        /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Using SSE2 crc32 instructions
es_1   | [2019-01-03T12:39:15,464][INFO ][o.e.n.Node               ] [SDWVkXe] stopping ...
api_1  | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc4204d9e38, 0x1a57380, 0xc42051a720, 0x1a57380, 0xc42051a720, 0xc4202cf150)
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
es_1   | [2019-01-03T12:39:15,568][INFO ][o.e.n.Node               ] [SDWVkXe] stopped
api_1  |        /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Completed initialization of buffer pool
es_1   | [2019-01-03T12:39:15,568][INFO ][o.e.n.Node               ] [SDWVkXe] closing ...
api_1  | github.com/trackit/trackit-server/aws/s3.init.1()
es_1   | [2019-01-03T12:39:15,581][INFO ][o.e.n.Node               ] [SDWVkXe] closed
sql_1  | 2019-01-03 12:39:05 140581952399104 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
es_1   | [2019-01-03T12:39:15,583][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
api_1  |        /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Highest supported file format is Barracuda.
api_1  | github.com/trackit/trackit-server/aws/s3.init()
api_1  |        <autogenerated>:1 +0x183
api_1  | github.com/trackit/trackit-server/aws/routes.init()
api_1  |        <autogenerated>:1 +0x8c
api_1  | main.init()
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: 128 out of 128 rollback segments are active.
trackit_es_1 exited with code 78
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Creating shared tablespace for temporary tables
api_1  |        <autogenerated>:1 +0x78
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: 5.7.19 started; log sequence number 1619987
sql_1  | 2019-01-03 12:39:05 140581711894272 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] Plugin 'FEEDBACK' is disabled.
sql_1  | 2019-01-03 12:39:05 140581711894272 [Note] InnoDB: Buffer pool(s) load completed at 190103 12:39:05
sql_1  | 2019-01-03 12:39:05 140582673598336 [Warning] 'user' entry 'root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-03 12:39:05 140582673598336 [Warning] 'proxies_priv' entry '@% root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] Reading of all Master_info entries succeded
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] Added new Master_info '' to hash table
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] mysqld: ready for connections.
sql_1  | Version: '10.2.9-MariaDB-10.2.9+maria~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
sql_1  | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
sql_1  | GENERATED ROOT PASSWORD: eefuthei9lohCuxohl9me0taegh0oini
sql_1  | 2019-01-03 12:39:09 140582537709312 [Warning] 'proxies_priv' entry '@% root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-03 12:39:09 140582537709312 [Warning] 'proxies_priv' entry '@% root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  |
sql_1  | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/schema.sql
sql_1  |
sql_1  |
sql_1  | 2019-01-03 12:39:11 140582537406208 [Note] mysqld (unknown): Normal shutdown
sql_1  |
sql_1  | 2019-01-03 12:39:11 140582537406208 [Note] Event Scheduler: Purging the queue. 0 events
sql_1  | 2019-01-03 12:39:11 140581829326592 [Note] InnoDB: FTS optimize thread exiting.
sql_1  | 2019-01-03 12:39:11 140582537406208 [Note] InnoDB: Starting shutdown...
sql_1  | 2019-01-03 12:39:11 140581711894272 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-03 12:39:11 140581711894272 [Note] InnoDB: Buffer pool(s) dump completed at 190103 12:39:11
sql_1  | 2019-01-03 12:39:13 140582537406208 [Note] InnoDB: Shutdown completed; log sequence number 2322244
sql_1  | 2019-01-03 12:39:13 140582537406208 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
sql_1  | 2019-01-03 12:39:13 140582537406208 [Note] mysqld: Shutdown complete
sql_1  |
sql_1  |
sql_1  | MySQL init process done. Ready for start up.
sql_1  |
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 1 ...
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
trackit_api_1 exited with code 2
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Uses event mutexes
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Compressed tables use zlib 1.2.8
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Using Linux native AIO
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Number of pools: 1
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Using SSE2 crc32 instructions
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Completed initialization of buffer pool
sql_1  | 2019-01-03 12:39:13 140656581883648 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Highest supported file format is Barracuda.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: 128 out of 128 rollback segments are active.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Creating shared tablespace for temporary tables
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Waiting for purge to start
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: 5.7.19 started; log sequence number 2322244
sql_1  | 2019-01-03 12:39:13 140656345343744 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] Plugin 'FEEDBACK' is disabled.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] Server socket created on IP: '::'.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Warning] 'proxies_priv' entry '@% root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] Reading of all Master_info entries succeded
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] Added new Master_info '' to hash table
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] mysqld: ready for connections.
sql_1  | Version: '10.2.9-MariaDB-10.2.9+maria~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
sql_1  | 2019-01-03 12:39:13 140656345343744 [Note] InnoDB: Buffer pool(s) load completed at 190103 12:39:13
sql_1  | 2019-01-03 12:41:07 140657165969152 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets)
sathyajackson commented 5 years ago

Please ignore my previous comment, I see the elastic search is not running. Let me fix that and update you with the latest logs.

sathyajackson commented 5 years ago

Here you go complete logs from all containers.

docker-compose logs -f
WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Attaching to trackit_api_1, trackit_sql_1, trackit_es_1, trackit_ui_1
es_1   | [2019-01-04T08:26:31,805][INFO ][o.e.n.Node               ] [] initializing ...
es_1   | [2019-01-04T08:26:31,886][INFO ][o.e.e.NodeEnvironment    ] [qxFs6OU] using [1] data paths, mounts [[/ (overlay)]], net usable_space [4.4gb], net total_space [29.4gb], spins? [possibly], types [overlay]
es_1   | [2019-01-04T08:26:31,887][INFO ][o.e.e.NodeEnvironment    ] [qxFs6OU] heap size [1.9gb], compressed ordinary object pointers [true]
es_1   | [2019-01-04T08:26:31,888][INFO ][o.e.n.Node               ] node name [qxFs6OU] derived from node ID [qxFs6OUqTVCOiQXlkExiig]; set [node.name] to override
es_1   | [2019-01-04T08:26:31,888][INFO ][o.e.n.Node               ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.77-70.59.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
es_1   | [2019-01-04T08:26:31,888][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [aggs-matrix-stats]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [ingest-common]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [lang-expression]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [lang-groovy]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [lang-mustache]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [lang-painless]
es_1   | [2019-01-04T08:26:36,662][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [parent-join]
es_1   | [2019-01-04T08:26:36,663][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [percolator]
es_1   | [2019-01-04T08:26:36,663][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [reindex]
es_1   | [2019-01-04T08:26:36,663][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [transport-netty3]
es_1   | [2019-01-04T08:26:36,663][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [transport-netty4]
es_1   | [2019-01-04T08:26:36,664][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded plugin [ingest-geoip]
es_1   | [2019-01-04T08:26:36,664][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded plugin [ingest-user-agent]
es_1   | [2019-01-04T08:26:36,664][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded plugin [x-pack]
es_1   | [2019-01-04T08:26:40,463][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/48] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
es_1   | [2019-01-04T08:26:40,522][INFO ][o.e.d.DiscoveryModule    ] [qxFs6OU] using discovery type [zen]
es_1   | [2019-01-04T08:26:41,559][INFO ][o.e.n.Node               ] initialized
es_1   | [2019-01-04T08:26:41,559][INFO ][o.e.n.Node               ] [qxFs6OU] starting ...
ui_1   | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
ui_1   |   'Supervisord is running as root and it is searching '
es_1   | [2019-01-04T08:26:41,822][INFO ][o.e.t.TransportService   ] [qxFs6OU] publish_address {10.19.84.3:9300}, bound_addresses {0.0.0.0:9300}
api_1  | {"level":"warning","time":"2019-01-04T08:26:29.248415991Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
api_1  | {"level":"warning","time":"2019-01-04T08:26:31.249527361Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
es_1   | [2019-01-04T08:26:41,833][INFO ][o.e.b.BootstrapChecks    ] [qxFs6OU] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ui_1   | 2019-01-04 08:26:28,881 CRIT Supervisor running as root (no user in config file)
sql_1  | Initializing database
api_1  | {"level":"warning","time":"2019-01-04T08:26:33.250291638Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
es_1   | [2019-01-04T08:26:44,905][INFO ][o.e.c.s.ClusterService   ] [qxFs6OU] new_master {qxFs6OU}{qxFs6OUqTVCOiQXlkExiig}{Aj1med4IQKiLrobsQj7XRg}{10.19.84.3}{10.19.84.3:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
ui_1   | 2019-01-04 08:26:28,920 INFO supervisord started with pid 1
sql_1  |
api_1  | {"level":"warning","time":"2019-01-04T08:26:35.2510975Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
es_1   | [2019-01-04T08:26:44,965][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [qxFs6OU] publish_address {10.19.84.3:9200}, bound_addresses {0.0.0.0:9200}
ui_1   | 2019-01-04 08:26:29,926 INFO spawned: 'webui' with pid 8
sql_1  | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
api_1  | {"level":"warning","time":"2019-01-04T08:26:37.252038723Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
es_1   | [2019-01-04T08:26:44,966][INFO ][o.e.n.Node               ] [qxFs6OU] started
ui_1   | 2019-01-04 08:26:31,710 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
sql_1  | To do so, start the server, then issue the following commands:
api_1  | {"level":"info","time":"2019-01-04T08:26:39.253442048Z","message":"Successfully connected to SQL database."}
es_1   | [2019-01-04T08:26:45,062][INFO ][o.e.g.GatewayService     ] [qxFs6OU] recovered [0] indices into cluster_state
sql_1  |
api_1  | {"level":"warning","time":"2019-01-04T08:26:44.409947762Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.3:9200: getsockopt: connection refused: no Elasticsearch node available"}
es_1   | [2019-01-04T08:26:45,379][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [qxFs6OU] successfully created .ml-notifications index template
sql_1  | '/usr/bin/mysqladmin' -u root password 'new-password'
api_1  | {"level":"info","time":"2019-01-04T08:26:46.676902103Z","message":"Successfully connected to ElasticSearch database."}
es_1   | [2019-01-04T08:26:45,464][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [qxFs6OU] successfully created .ml-state index template
sql_1  | '/usr/bin/mysqladmin' -u root -h  password 'new-password'
api_1  | {"level":"info","time":"2019-01-04T08:26:46.720897383Z","message":"Put ES index lineitems.","data":{"acknowledged":true}}
es_1   | [2019-01-04T08:26:45,487][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [qxFs6OU] successfully created .ml-meta index template
sql_1  |
api_1  | {"level":"info","time":"2019-01-04T08:26:46.759298144Z","message":"Put ES index AnomaliesDetection.","data":{"acknowledged":true}}
es_1   | [2019-01-04T08:26:45,534][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [qxFs6OU] successfully created .ml-anomalies- index template
es_1   | [2019-01-04T08:26:45,783][INFO ][o.e.l.LicenseService     ] [qxFs6OU] license [b117f115-3038-4f87-bcea-817a1c3945cd] mode [trial] - valid
sql_1  | Alternatively you can run:
api_1  | {"level":"info","time":"2019-01-04T08:26:46.783406905Z","message":"Put ES index EC2Report.","data":{"acknowledged":true}}
sql_1  | '/usr/bin/mysql_secure_installation'
sql_1  |
api_1  | {"level":"info","time":"2019-01-04T08:26:46.80606293Z","message":"Put ES index EC2 Coverage Report.","data":{"acknowledged":true}}
sql_1  | which will also give you the option of removing the test
sql_1  | databases and anonymous user created by default.  This is
api_1  | {"level":"info","time":"2019-01-04T08:26:46.82978386Z","message":"Put ES index ElastiCache Report.","data":{"acknowledged":true}}
sql_1  | strongly recommended for production servers.
sql_1  |
sql_1  | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
api_1  | {"level":"info","time":"2019-01-04T08:26:46.853230022Z","message":"Put ES index ESReport.","data":{"acknowledged":true}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.875109694Z","message":"Put ES index rds-reports.","data":{"acknowledged":true}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.896676091Z","message":"Put ES index account-plugins.","data":{"acknowledged":true}}
sql_1  | MySQL manual for more instructions.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.919304328Z","message":"Put ES index LambdaReport.","data":{"acknowledged":true}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.942740832Z","message":"Put ES index ReservedInstancesReport.","data":{"acknowledged":true}}
sql_1  |
api_1  | {"level":"info","time":"2019-01-04T08:26:46.964842393Z","message":"Put ES index rds-ri-reports.","data":{"acknowledged":true}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965185801Z","message":"Started.","data":{"backendId":"c5bd6e79d172-e1b2acaf"}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965366838Z","message":"Registered route /user."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965454595Z","message":"Registered route /user/viewer."}
sql_1  | Please report any problems at http://mariadb.org/jira
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965534201Z","message":"Registered route /user/login."}
sql_1  |
sql_1  | The latest information about MariaDB is available at http://mariadb.org/.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.96561251Z","message":"Registered route /user/password/forgotten."}
sql_1  | You can find additional information about the MySQL part at:
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965694603Z","message":"Registered route /user/password/reset."}
sql_1  | http://dev.mysql.com
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965773453Z","message":"Registered route /aws/billrepository."}
sql_1  | Consider joining MariaDB's strong and vibrant community:
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965851138Z","message":"Registered route /aws/billrepositoryupdates."}
sql_1  | https://mariadb.org/get-involved/
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965924772Z","message":"Registered route /aws."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966009327Z","message":"Registered route /aws/subaccount."}
sql_1  |
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966068011Z","message":"Registered route /aws/next."}
sql_1  | Database initialized
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966158534Z","message":"Registered route /aws/status."}
sql_1  | MySQL init process in progress...
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966233233Z","message":"Registered route /costs."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966290696Z","message":"Registered route /costs/anomalies."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 97 ...
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966382513Z","message":"Registered route /ec2."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966434998Z","message":"Registered route /ec2/unused."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966523785Z","message":"Registered route /costs/diff."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966604845Z","message":"Registered route /costs/tags/values."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966656487Z","message":"Registered route /costs/tags/keys."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Uses event mutexes
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966724751Z","message":"Registered route /plugins/results."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966806776Z","message":"Registered route /elasticache."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Compressed tables use zlib 1.2.8
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Using Linux native AIO
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966865929Z","message":"Registered route /elasticache/unused."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Number of pools: 1
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966952045Z","message":"Registered route /es."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Using SSE2 crc32 instructions
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967031861Z","message":"Registered route /es/unused."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Completed initialization of buffer pool
sql_1  | 2019-01-04  8:26:31 139952870475520 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Highest supported file format is Barracuda.
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: 128 out of 128 rollback segments are active.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967078057Z","message":"Registered route /lambda."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Creating shared tablespace for temporary tables
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967157257Z","message":"Registered route /rds."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Waiting for purge to start
api_1  | {"level":"info","time":"2019-01-04T08:26:46.96723126Z","message":"Registered route /rds/unused."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967309101Z","message":"Registered route /reports."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967367537Z","message":"Registered route /report."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: 5.7.19 started; log sequence number 1619987
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967447021Z","message":"Registered route /s3/costs."}
sql_1  | 2019-01-04  8:26:31 139952364967680 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967512934Z","message":"Registered route /ec2/coverage."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] Plugin 'FEEDBACK' is disabled.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.9675619Z","message":"Registered route /ri/ec2."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967640239Z","message":"Registered route /ri/rds."}
sql_1  | 2019-01-04  8:26:31 139952364967680 [Note] InnoDB: Buffer pool(s) load completed at 190104  8:26:31
api_1  | {"level":"info","time":"2019-01-04T08:26:46.96770609Z","message":"Registered route /user/share."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967755306Z","message":"Registered route /docs."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Warning] 'user' entry 'root@fba613fdadad' ignored in --skip-name-resolve mode.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967839901Z","message":"Scheduled periodic tasks."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967910918Z","message":"Listening on [::]:80."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Warning] 'proxies_priv' entry '@% root@fba613fdadad' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] Reading of all Master_info entries succeded
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] Added new Master_info '' to hash table
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] mysqld: ready for connections.
sql_1  | Version: '10.2.9-MariaDB-10.2.9+maria~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
sql_1  | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
sql_1  | GENERATED ROOT PASSWORD: quo9Nu8UCh8ahgeedaeL8moo7azek8uw
sql_1  | 2019-01-04  8:26:35 139952956409600 [Warning] 'proxies_priv' entry '@% root@fba613fdadad' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-04  8:26:35 139952956409600 [Warning] 'proxies_priv' entry '@% root@fba613fdadad' ignored in --skip-name-resolve mode.
sql_1  |
sql_1  | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/schema.sql
sql_1  |
sql_1  |
sql_1  | 2019-01-04  8:26:37 139952956106496 [Note] mysqld (unknown): Normal shutdown
sql_1  |
sql_1  | 2019-01-04  8:26:37 139952956106496 [Note] Event Scheduler: Purging the queue. 0 events
sql_1  | 2019-01-04  8:26:37 139952750835456 [Note] InnoDB: FTS optimize thread exiting.
sql_1  | 2019-01-04  8:26:37 139952956106496 [Note] InnoDB: Starting shutdown...
sql_1  | 2019-01-04  8:26:37 139952364967680 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-04  8:26:37 139952364967680 [Note] InnoDB: Buffer pool(s) dump completed at 190104  8:26:37
sql_1  | 2019-01-04  8:26:38 139952956106496 [Note] InnoDB: Shutdown completed; log sequence number 2323307
sql_1  | 2019-01-04  8:26:38 139952956106496 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
sql_1  | 2019-01-04  8:26:38 139952956106496 [Note] mysqld: Shutdown complete
sql_1  |
sql_1  |
sql_1  | MySQL init process done. Ready for start up.
sql_1  |
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 1 ...
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Uses event mutexes
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Compressed tables use zlib 1.2.8
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Using Linux native AIO
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Number of pools: 1
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Using SSE2 crc32 instructions
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Completed initialization of buffer pool
sql_1  | 2019-01-04  8:26:38 140426037094144 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Highest supported file format is Barracuda.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: 128 out of 128 rollback segments are active.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: Creating shared tablespace for temporary tables
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: Waiting for purge to start
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: 5.7.19 started; log sequence number 2323307
sql_1  | 2019-01-04  8:26:39 140425868326656 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] Plugin 'FEEDBACK' is disabled.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] Server socket created on IP: '::'.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Warning] 'proxies_priv' entry '@% root@fba613fdadad' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] Reading of all Master_info entries succeded
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] Added new Master_info '' to hash table
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] mysqld: ready for connections.
sql_1  | Version: '10.2.9-MariaDB-10.2.9+maria~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
sql_1  | 2019-01-04  8:26:39 140425868326656 [Note] InnoDB: Buffer pool(s) load completed at 190104  8:26:39
es_1   | [2019-01-04T08:26:51,780][INFO ][o.e.c.m.MetaDataCreateIndexService] [qxFs6OU] [.monitoring-es-6-2019.01.04] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
es_1   | [2019-01-04T08:26:51,946][INFO ][o.e.c.m.MetaDataCreateIndexService] [qxFs6OU] [.watches] creating index, cause [auto(bulk api)], templates [watches], shards [1]/[1], mappings [watch]
es_1   | [2019-01-04T08:26:52,173][INFO ][o.e.c.m.MetaDataMappingService] [qxFs6OU] [.watches/xXa5S8_cTD2ebHqqF3fo-w] update_mapping [watch]
es_1   | [2019-01-04T08:27:14,940][INFO ][o.e.c.r.a.DiskThresholdMonitor] [qxFs6OU] low disk watermark [85%] exceeded on [qxFs6OUqTVCOiQXlkExiig][qxFs6OU][/usr/share/elasticsearch/data/nodes/0] free: 4.3gb[14.8%], replicas will not be assigned to this node

Here is the running containers.

ocker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                      NAMES
c5bd6e79d172        trackit_api                                           "/main -sql-address=…"   3 minutes ago       Up 2 minutes        0.0.0.0:8080->80/tcp                       trackit_api_1
fba613fdadad        trackit_sql                                           "docker-entrypoint.s…"   3 minutes ago       Up 3 minutes        127.0.0.1:3306->3306/tcp                   trackit_sql_1
e31ff5ebc91d        docker.elastic.co/elasticsearch/elasticsearch:5.5.3   "/bin/bash bin/es-do…"   3 minutes ago       Up 3 minutes        127.0.0.1:9200->9200/tcp, 9300/tcp         trackit_es_1
ea0e400bc3e1        trackit_ui                                            "/usr/bin/supervisord"   3 minutes ago       Up 3 minutes        0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   trackit_ui_1

Still i am having the same problem, while registering, login or even the forgot password link.

thibautcornolti commented 5 years ago

I see you still have these errors

WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.

For now, we'll ignore them, to understand why you can't login.

Go to http://127.0.0.1/, you'll see the login page. Then, display the network page in the developer tools (Press F12 or Ctrl + Shift + I). Try to create an account. You'll see a request with an error, write here the details.

sathyajackson commented 5 years ago

The below snip is what i see when i try to register a new user using my gmail email address

image

thibautcornolti commented 5 years ago

Please click on the last one and screen again

sathyajackson commented 5 years ago

I think i got the issue sorted out. It was beacuse of a proxy server in the network which was causing the problem. I sorted it out and now i can able to register.

But why am i still getting this warning.


[ warning ]

WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string. WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string. WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string. WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.

thibautcornolti commented 5 years ago

If

I'm not sure why you're still getting this

sathyajackson commented 5 years ago

Just want to understand, why we need to provide read-only access to an external account to have this working?

thibautcornolti commented 5 years ago

TrackIt needs an AWS account to assume roles (https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Roles are asked in the UI and used to fetch bills and other data from the AWS account you want to monitor.