search

telekom-security / tpotce

🍯 T-Pot - The All In One Multi Honeypot Platform 🐝
GNU General Public License v3.0
6.82k stars 1.08k forks source link

Saved data after update #338

Closed mahmoodn closed 5 years ago

mahmoodn commented 5 years ago

I think I asked a question but can not find that! So sorry if this is a duplicated one.... Thanks for the latest release, 19.03. I want to know is it OK to reinstall the iso? What will happen to the current saved data? Surely, we get a backup first, but I want to know if current data remains consistent after upgrade.

t3chn0m4g3 commented 5 years ago

There is no automatic backup and there is no guarantee that export / import of elastic data will be consistent accross versions. However you can export all elastic data with <tpotroot>/bin/dump_es.sh, but make sure you use the one from the master branch. On the new machine you may import the data with <tpotroot>/bin/restore_es.sh. This will probably overwrite the Kibana settings for the new install as well and you must import the kibana objects with import_kibana-objects.sh /opt/tpot/etc/objects/kibana-objects.tgz.

t3chn0m4g3 commented 5 years ago

Thanks for the feedback and the appreciation.

mahmoodn commented 5 years ago

Sure. I will try that. Thanks.

mahmoodn commented 5 years ago

OK. It seems that I have to reopen the thread. In the restore command on 19.03, I hit this:

Sun, 07 Apr 2019 18:37:59 GMT | sent 7551 objects to destination elasticsearch, wrote 7551
Sun, 07 Apr 2019 18:37:59 GMT | got 5068 objects from source file (offset: 67818)
Sun, 07 Apr 2019 18:38:01 GMT | sent 5068 objects to destination elasticsearch, wrote 5068
Sun, 07 Apr 2019 18:38:01 GMT | got 0 objects from source file (offset: 72886)
Sun, 07 Apr 2019 18:38:01 GMT | Total Writes: 72886
Sun, 07 Apr 2019 18:38:01 GMT | dump complete
### Now uncompressing: tmp/.kibana.gz
gzip: tmp/.kibana.gz: No such file or directory
### Now restoring: .kibana
Sun, 07 Apr 2019 18:38:02 GMT | starting dump
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: ENOENT: no such file or directory, open 'tmp/.kibana'
Emitted 'error' event at:
    at lazyFs.open (internal/fs/streams.js:115:12)
    at FSReqWrap.oncomplete (fs.js:141:20)
rm: cannot remove 'tmp/.kibana': No such file or directory
### Done.

The tar file contains

[root@successivetinkle:/home/tsec]# tar -tvf es_dump_201904070605.tar
drwxr-xr-x root/root         0 2019-04-07 06:43 tmp/./
-rw-r--r-- root/root   6595896 2019-04-07 06:22 tmp/./logstash-2019.03.11.gz
-rw-r--r-- root/root   9412878 2019-04-07 06:17 tmp/./logstash-2019.02.27.gz
-rw-r--r-- root/root  34168221 2019-04-07 06:43 tmp/./.monitoring-es-6-2019.04.07.gz
-rw-r--r-- root/root  10436367 2019-04-07 06:26 tmp/./logstash-2019.03.25.gz
-rw-r--r-- root/root   5584504 2019-04-07 06:22 tmp/./logstash-2019.03.12.gz
-rw-r--r-- root/root   9639020 2019-04-07 06:28 tmp/./logstash-2019.03.28.gz
-rw-r--r-- root/root   4286149 2019-04-07 06:26 tmp/./logstash-2019.03.23.gz
-rw-r--r-- root/root   7414507 2019-04-07 06:09 tmp/./logstash-2019.02.07.gz
-rw-r--r-- root/root  12998147 2019-04-07 06:10 tmp/./logstash-2019.02.10.gz
-rw-r--r-- root/root   7121517 2019-04-07 06:22 tmp/./logstash-2019.03.10.gz
-rw-r--r-- root/root   6313346 2019-04-07 06:23 tmp/./logstash-2019.03.14.gz
-rw-r--r-- root/root  13655000 2019-04-07 06:14 tmp/./logstash-2019.02.22.gz
-rw-r--r-- root/root   6586783 2019-04-07 06:12 tmp/./logstash-2019.02.16.gz
-rw-r--r-- root/root   8104774 2019-04-07 06:25 tmp/./logstash-2019.03.21.gz
-rw-r--r-- root/root   6330786 2019-04-07 06:12 tmp/./logstash-2019.02.14.gz
-rw-r--r-- root/root      2691 2019-04-07 06:13 tmp/./logstash-2019.02.18.gz
-rw-r--r-- root/root  14419625 2019-04-07 06:27 tmp/./logstash-2019.03.26.gz
-rw-r--r-- root/root  22970525 2019-04-07 06:19 tmp/./logstash-2019.03.03.gz
-rw-r--r-- root/root   7743972 2019-04-07 06:24 tmp/./logstash-2019.03.18.gz
-rw-r--r-- root/root   5799494 2019-04-07 06:11 tmp/./logstash-2019.02.11.gz
-rw-r--r-- root/root   6432280 2019-04-07 06:13 tmp/./logstash-2019.02.17.gz
-rw-r--r-- root/root  10863894 2019-04-07 06:15 tmp/./logstash-2019.02.25.gz
-rw-r--r-- root/root  13243141 2019-04-07 06:18 tmp/./logstash-2019.03.02.gz
-rw-r--r-- root/root   9333966 2019-04-07 06:10 tmp/./logstash-2019.02.09.gz
-rw-r--r-- root/root  32843953 2019-04-07 06:36 tmp/./.monitoring-es-6-2019.04.02.gz
-rw-r--r-- root/root  11220218 2019-04-07 06:29 tmp/./logstash-2019.03.31.gz
-rw-r--r-- root/root   3258693 2019-04-07 06:05 tmp/./logstash-2019.01.26.gz
-rw-r--r-- root/root   8185930 2019-04-07 06:29 tmp/./logstash-2019.04.01.gz
-rw-r--r-- root/root   6186429 2019-04-07 06:09 tmp/./logstash-2019.02.08.gz
-rw-r--r-- root/root  13804442 2019-04-07 06:14 tmp/./logstash-2019.02.23.gz
-rw-r--r-- root/root   4542651 2019-04-07 06:09 tmp/./logstash-2019.02.06.gz
-rw-r--r-- root/root   8693775 2019-04-07 06:23 tmp/./logstash-2019.03.15.gz
-rw-r--r-- root/root   5997249 2019-04-07 06:21 tmp/./logstash-2019.03.09.gz
-rw-r--r-- root/root   5287053 2019-04-07 06:13 tmp/./logstash-2019.02.20.gz
-rw-r--r-- root/root   3467824 2019-04-07 06:30 tmp/./logstash-2019.04.07.gz
-rw-r--r-- root/root   7173329 2019-04-07 06:24 tmp/./logstash-2019.03.17.gz
-rw-r--r-- root/root   6522812 2019-04-07 06:09 tmp/./logstash-2019.02.05.gz
-rw-r--r-- root/root   9007076 2019-04-07 06:28 tmp/./logstash-2019.03.29.gz
-rw-r--r-- root/root   7033176 2019-04-07 06:23 tmp/./logstash-2019.03.13.gz
-rw-r--r-- root/root 112867552 2019-04-07 06:40 tmp/./.monitoring-es-6-2019.04.06.gz
-rw-r--r-- root/root    245138 2019-04-07 06:43 tmp/./.monitoring-kibana-6-2019.04.07.gz
-rw-r--r-- root/root   6432440 2019-04-07 06:07 tmp/./logstash-2019.01.30.gz
-rw-r--r-- root/root   6899535 2019-04-07 06:30 tmp/./logstash-2019.04.02.gz
-rw-r--r-- root/root      1585 2019-04-07 06:13 tmp/./logstash-2019.02.19.gz
-rw-r--r-- root/root   4551205 2019-04-07 06:07 tmp/./logstash-2019.01.31.gz
-rw-r--r-- root/root  10879352 2019-04-07 06:27 tmp/./logstash-2019.03.27.gz
-rw-r--r-- root/root   6027721 2019-04-07 06:21 tmp/./logstash-2019.03.07.gz
-rw-r--r-- root/root   3900714 2019-04-07 06:26 tmp/./logstash-2019.03.22.gz
-rw-r--r-- root/root    773731 2019-04-07 06:43 tmp/./.monitoring-kibana-6-2019.04.06.gz
-rw-r--r-- root/root   6188124 2019-04-07 06:07 tmp/./logstash-2019.02.01.gz
-rw-r--r-- root/root  11484140 2019-04-07 06:18 tmp/./logstash-2019.03.01.gz
-rw-r--r-- root/root    846440 2019-04-07 06:05 tmp/./kibana_sample_data_ecommerce.gz
-rw-r--r-- root/root  26022478 2019-04-07 06:08 tmp/./logstash-2019.02.02.gz
-rw-r--r-- root/root    323175 2019-04-07 06:43 tmp/./.monitoring-kibana-6-2019.04.01.gz
-rw-r--r-- root/root  10248841 2019-04-07 06:06 tmp/./logstash-2019.01.29.gz
-rw-r--r-- root/root   3124256 2019-04-07 06:06 tmp/./logstash-2019.01.28.gz
-rw-r--r-- root/root   5653876 2019-04-07 06:12 tmp/./logstash-2019.02.15.gz
-rw-r--r-- root/root   6127583 2019-04-07 06:20 tmp/./logstash-2019.03.04.gz
-rw-r--r-- root/root   7530258 2019-04-07 06:25 tmp/./logstash-2019.03.20.gz
-rw-r--r-- root/root   8930928 2019-04-07 06:11 tmp/./logstash-2019.02.12.gz
-rw-r--r-- root/root  45579673 2019-04-07 06:34 tmp/./.monitoring-es-6-2019.04.01.gz
-rw-r--r-- root/root   6898853 2019-04-07 06:11 tmp/./logstash-2019.02.13.gz
-rw-r--r-- root/root  44037613 2019-04-07 06:32 tmp/./.monitoring-es-6-2019.03.31.gz
-rw-r--r-- root/root   9734056 2019-04-07 06:15 tmp/./logstash-2019.02.24.gz
-rw-r--r-- root/root  13827816 2019-04-07 06:21 tmp/./logstash-2019.03.06.gz
-rw-r--r-- root/root  15844080 2019-04-07 06:30 tmp/./logstash-2019.04.06.gz
-rw-r--r-- root/root   2141340 2019-04-07 06:21 tmp/./logstash-2019.03.08.gz
-rw-r--r-- root/root    316190 2019-04-07 06:43 tmp/./.monitoring-kibana-6-2019.03.31.gz
-rw-r--r-- root/root   2277010 2019-04-07 06:43 tmp/./.reporting-2019.01.27.gz
-rw-r--r-- root/root   9800518 2019-04-07 06:13 tmp/./logstash-2019.02.21.gz
-rw-r--r-- root/root    228596 2019-04-07 06:43 tmp/./.monitoring-kibana-6-2019.04.02.gz
-rw-r--r-- root/root   1535000 2019-04-07 06:05 tmp/./kibana_sample_data_logs.gz
-rw-r--r-- root/root  13641829 2019-04-07 06:20 tmp/./logstash-2019.03.05.gz
-rw-r--r-- root/root  10862041 2019-04-07 06:29 tmp/./logstash-2019.03.30.gz
-rw-r--r-- root/root   9543170 2019-04-07 06:25 tmp/./logstash-2019.03.19.gz
-rw-r--r-- root/root   1323138 2019-04-07 06:08 tmp/./logstash-2019.02.04.gz
-rw-r--r-- root/root  20815748 2019-04-07 06:16 tmp/./logstash-2019.02.26.gz
-rw-r--r-- root/root  11507950 2019-04-07 06:06 tmp/./logstash-2019.01.27.gz
-rw-r--r-- root/root     49039 2019-04-07 06:05 tmp/./.kibana_1.gz
-rw-r--r-- root/root  13978285 2019-04-07 06:17 tmp/./logstash-2019.02.28.gz
-rw-r--r-- root/root   3283970 2019-04-07 06:26 tmp/./logstash-2019.03.24.gz
-rw-r--r-- root/root   6613071 2019-04-07 06:24 tmp/./logstash-2019.03.16.gz
-rw-r--r-- root/root  27309344 2019-04-07 06:08 tmp/./logstash-2019.02.03.gz
[root@successivetinkle:/home/tsec]#

It seems that it is expecting Now uncompressing: tmp/.kibana.gz but the file in the back up is tmp/./.kibana_1.gz. This is my guess. Any idea? Thanks.

t3chn0m4g3 commented 5 years ago

Did you use the scripts from the master branch as suggested?

t3chn0m4g3 commented 5 years ago

Assuming this is resolved.

mahmoodn commented 5 years ago

Hi. Sorry for the delay. We had some networking problems and tpot wasn't consistent. Still we have some network problems. I actually missed the master branch which you said. So, I downloaded that and ran dump_es.sh on 18.11. After copying the file to 19.03 machine, I ran restore_es.sh and it finished without any error. Full log is shown here.

Then I imported kibana objects as below.

[tsec@experimentalovercharge:~]$ import_kibana-objects.sh /opt/tpot/etc/objects/kibana-objects.tgz
### Elasticsearch is available, now continuing.

### Now importing 943 index pattern fields.

### Now importing 19 dashboards.
###### 14ebefd0-488f-11e8-9b3d-f36e8d4f5cb2
###### 48556a20-3f58-11e9-8971-b9d98cc69287
###### 48f595c0-e7f8-11e8-9ac4-13ecd4ad8d70
###### 54d8c6a0-dec0-11e8-87cf-239397d2b8d3
###### 7e1f5930-465c-11e9-9d82-ab3a5e06df4c
###### 83b11b50-e850-11e8-97df-bbc3de28ece0
###### 891769a0-fa2f-11e8-838f-fff066e21110
###### 8d4e8300-ebde-11e8-9675-1b303bfb38ef
###### ConPot
###### Cowrie
###### dd95c950-8b5d-11e7-ba35-0d8832ac304f
###### Dionaea
###### e05aac20-8b51-11e7-b92d-d39e43e3de0f
###### ElasticPot
###### faeb1340-6355-11e8-be86-73985bedf977
###### Glastopf
###### Honeytrap
###### NGINX
###### Suricata

### Now importing 255 visualizations.
###### 0169b450-8b62-11e7-ba35-0d8832ac304f
###### 0464b030-e852-11e8-97df-bbc3de28ece0
###### 051c59e0-8b53-11e7-b92d-d39e43e3de0f
###### 06628c70-6352-11e8-be86-73985bedf977
###### 07581df0-e752-11e8-b4a6-215b0b97c069
###### 082111a0-80cf-11e7-ab37-eb92b1bfb573
###### 099c23d0-e7fd-11e8-9ac4-13ecd4ad8d70
###### 0d947000-7ebd-11e7-a286-9f03beba6417
###### 0de24040-8b52-11e7-b92d-d39e43e3de0f
###### 0e230290-859b-11e7-8f60-4f4666b0a88e
###### 0f9b72a0-fa30-11e8-838f-fff066e21110
###### 0fe996f0-5085-11e9-84e8-4f3cd8f0c2c8
###### 106aec70-4b15-11e9-94b3-a1bc5949ac31
###### 10e765a0-e51e-11e8-8a75-d5f374dbaebe
###### 124a1140-488e-11e8-9b3d-f36e8d4f5cb2
###### 1268af10-4889-11e8-9b3d-f36e8d4f5cb2
###### 133801c0-e852-11e8-97df-bbc3de28ece0
###### 15b696f0-e7fd-11e8-9ac4-13ecd4ad8d70
###### 15f2c000-4ad5-11e8-ab1b-fdef76c312f4
###### 160f9cb0-e85b-11e8-97df-bbc3de28ece0
###### 17130490-e7fb-11e8-9ac4-13ecd4ad8d70
###### 1a097850-7c22-11e7-aa1e-6bf93670d67b
###### 1a80b720-4ad6-11e8-ab1b-fdef76c312f4
###### 20d12bf0-fa35-11e8-838f-fff066e21110
###### 21ad1c80-488a-11e8-9b3d-f36e8d4f5cb2
###### 21c65b10-e852-11e8-97df-bbc3de28ece0
###### 29e9d050-465e-11e9-9d82-ab3a5e06df4c
###### 29f51af0-4876-11e8-9b3d-f36e8d4f5cb2
###### 2a543aa0-4ad6-11e8-ab1b-fdef76c312f4
###### 2a6803f0-80e7-11e7-a689-67e589a14a8a
###### 2cf90930-47d3-11e8-a905-f74bbc7cbd2d
###### 2fc62270-e872-11e8-a876-6bccfb9086f2
###### 307afd60-82a9-11e7-bcbe-2b6958a9c888
###### 32814dd0-e851-11e8-97df-bbc3de28ece0
###### 349c11c0-7ea0-11e7-a286-9f03beba6417
###### 3814c570-e68e-11e8-b727-735f5b0e1502
###### 39156290-4b15-11e9-94b3-a1bc5949ac31
###### 3f646820-e851-11e8-97df-bbc3de28ece0
###### 409907c0-e6b5-11e8-b727-735f5b0e1502
###### 41d04290-e852-11e8-97df-bbc3de28ece0
###### 45e32dc0-dec5-11e8-87cf-239397d2b8d3
###### 465d9810-e5cf-11e8-b72a-b734d2b55cd4
###### 48d3dad0-e725-11e8-b4a6-215b0b97c069
###### 490b4e60-e7fd-11e8-9ac4-13ecd4ad8d70
###### 4a60fe20-e75f-11e8-803c-59c072645505
###### 4db00710-465f-11e9-9d82-ab3a5e06df4c
###### 4e2887d0-8379-11e7-97dc-15d31af3c77f
###### 5014cee0-634e-11e8-be86-73985bedf977
###### 50aa1940-e851-11e8-97df-bbc3de28ece0
###### 50d82860-7ea0-11e7-a286-9f03beba6417
###### 51c331f0-8b54-11e7-b92d-d39e43e3de0f
###### 51ca6ee0-80d5-11e7-ab37-eb92b1bfb573
###### 5234de80-8b5f-11e7-b92d-d39e43e3de0f
###### 535b0c80-e761-11e8-803c-59c072645505
###### 54213440-8b56-11e7-b92d-d39e43e3de0f
###### 554c9550-e7fb-11e8-9ac4-13ecd4ad8d70
###### 56cdedf0-ec08-11e8-96db-ebfb2a58ccf6
###### 576a3cb0-82ae-11e7-bcbe-2b6958a9c888
###### 59509e90-8590-11e7-a686-392ac617767d
###### 59b9dd60-827f-11e7-afbf-a7491fba5d8a
###### 5b081fc0-465e-11e9-9d82-ab3a5e06df4c
###### 5b69e990-e7fd-11e8-9ac4-13ecd4ad8d70
###### 5c468140-4660-11e9-9d82-ab3a5e06df4c
###### 621f1ae0-fa30-11e8-838f-fff066e21110
###### 62efe620-fa35-11e8-838f-fff066e21110
###### 62fde9a0-858d-11e7-a686-392ac617767d
###### 63672eb0-8b5f-11e7-b92d-d39e43e3de0f
###### 63d0bf60-e851-11e8-97df-bbc3de28ece0
###### 641f0c50-465d-11e9-9d82-ab3a5e06df4c
###### 651915d0-4b15-11e9-94b3-a1bc5949ac31
###### 656df650-6357-11e8-be86-73985bedf977
###### 65fdfd10-e7f8-11e8-9ac4-13ecd4ad8d70
###### 6ee57da0-634f-11e8-be86-73985bedf977
###### 6ee70b90-8374-11e7-9adb-2955c2136c8c
###### 73364660-8b54-11e7-b92d-d39e43e3de0f
###### 762f66c0-7e9e-11e7-a286-9f03beba6417
###### 772cb2b0-7ec3-11e7-a7c8-5f38ad5bf75f
###### 77bf1310-634e-11e8-be86-73985bedf977
###### 79399a60-4b17-11e9-94b3-a1bc5949ac31
###### 79a71e90-dec1-11e8-87cf-239397d2b8d3
###### 7b61a6a0-7ebf-11e7-a286-9f03beba6417
###### 7c5959b0-4889-11e8-9b3d-f36e8d4f5cb2
###### 7dcaa2b0-8596-11e7-a686-392ac617767d
###### 7e33e3d0-810c-11e7-8413-9fe5e30ade77
###### 7e6121d0-e851-11e8-97df-bbc3de28ece0
###### 7e9a7d20-e858-11e8-97df-bbc3de28ece0
###### 81840ef0-5097-11e9-8edb-e10ede9f8803
###### 844f33f0-488a-11e8-9b3d-f36e8d4f5cb2
###### 864b2f30-4883-11e8-9b3d-f36e8d4f5cb2
###### 87428ba0-7e9d-11e7-a286-9f03beba6417
###### 874be060-8b54-11e7-b92d-d39e43e3de0f
###### 87cf3b50-6357-11e8-be86-73985bedf977
###### 885928c0-7ebe-11e7-a286-9f03beba6417
###### 88d899e0-8b5f-11e7-b92d-d39e43e3de0f
###### 895645f0-6356-11e8-be86-73985bedf977
###### 8a455850-4ad5-11e8-ab1b-fdef76c312f4
###### 8b2db750-465e-11e9-9d82-ab3a5e06df4c
###### 8eb51e70-4660-11e9-9d82-ab3a5e06df4c
###### 946dc4d0-6352-11e8-be86-73985bedf977
###### 948a7490-465f-11e9-9d82-ab3a5e06df4c
###### 94ae10e0-4871-11e8-9b3d-f36e8d4f5cb2
###### 94e13130-e756-11e8-b4a6-215b0b97c069
###### 9881aac0-4b12-11e9-94b3-a1bc5949ac31
###### 9b89ddb0-ec07-11e8-96db-ebfb2a58ccf6
###### 9d251bd0-e851-11e8-97df-bbc3de28ece0
###### a001a350-e85b-11e8-97df-bbc3de28ece0
###### a204f440-465c-11e9-9d82-ab3a5e06df4c
###### a3500f20-fa2f-11e8-838f-fff066e21110
###### a427e6c0-e521-11e8-8a75-d5f374dbaebe
###### a51e9ae0-6350-11e8-be86-73985bedf977
###### a6ccd530-6352-11e8-be86-73985bedf977
###### a72ec5f0-4ad5-11e8-ab1b-fdef76c312f4
###### a81cba70-e7fc-11e8-9ac4-13ecd4ad8d70
###### abace4c0-4b12-11e9-94b3-a1bc5949ac31
###### acd09250-4b14-11e9-94b3-a1bc5949ac31
###### ad6dcb50-8b5e-11e7-ba35-0d8832ac304f
###### b1a7f8d0-859b-11e7-8f60-4f4666b0a88e
###### b1fcd330-465d-11e9-9d82-ab3a5e06df4c
###### b32152a0-4b16-11e9-94b3-a1bc5949ac31
###### b8745000-4ad5-11e8-ab1b-fdef76c312f4
###### b9343070-80e9-11e7-a689-67e589a14a8a
###### ba9d6280-8b5f-11e7-b92d-d39e43e3de0f
###### bc687e50-4b12-11e9-94b3-a1bc5949ac31
###### bd4055d0-4b14-11e9-94b3-a1bc5949ac31
###### bd5e53d0-fa30-11e8-838f-fff066e21110
###### bf39e000-80d5-11e7-ba6f-4542711dd148
###### bf6f6000-8598-11e7-8f60-4f4666b0a88e
###### c0916430-8b5e-11e7-ba35-0d8832ac304f
###### c1c8a3a0-6352-11e8-be86-73985bedf977
###### c1ef91c0-7dc2-11e7-8268-ed048f6272e0
###### c6bdc100-4660-11e9-9d82-ab3a5e06df4c
###### c80e9ff0-e866-11e8-95af-236f09a02fdb
###### c8e83c30-fa2f-11e8-838f-fff066e21110
###### c90f1f00-8b52-11e7-b92d-d39e43e3de0f
###### cac48440-8b5f-11e7-b92d-d39e43e3de0f
###### cb2a3a00-8b56-11e7-b92d-d39e43e3de0f
###### cf8d0e40-80ea-11e7-a689-67e589a14a8a
###### cfa9fb70-465c-11e9-9d82-ab3a5e06df4c
###### ConPot-ASN-Top-10
###### ConPot-Countries-Top-10
###### ConPot-Events-by-Country-Histogram
###### ConPot-Events-Histogram
###### ConPot-Event-Type
###### ConPot-Input-Top-10
###### ConPot-Map
###### ConPot-Protocol
###### ConPot-Response-Top-10
###### Cowrie-ASN-Top-10
###### Cowrie-Countries-Top-10
###### Cowrie-Events-by-Country-Histogram
###### Cowrie-Events-Histogram
###### Cowrie-Input-Top-10
###### Cowrie-Map
###### Cowrie-Password-Tagcloud-Large
###### Cowrie-Ports-Pie
###### Cowrie-Source-IP-Top-10
###### Cowrie-Username-Tagcloud-Large
###### Cowrie-Version-Pie-Top-10
###### d01a6390-827e-11e7-afbf-a7491fba5d8a
###### d0dbe890-4870-11e8-9b3d-f36e8d4f5cb2
###### d1aa9740-7e9e-11e7-a286-9f03beba6417
###### d224ffa0-4b14-11e9-94b3-a1bc5949ac31
###### d2405e70-8b5e-11e7-ba35-0d8832ac304f
###### d3bb9bd0-4863-11e8-9b3d-f36e8d4f5cb2
###### d500a3c0-e6b8-11e8-b727-735f5b0e1502
###### d77bbba0-4ad5-11e8-ab1b-fdef76c312f4
###### d94ff2a0-7ec2-11e7-a286-9f03beba6417
###### d968d5e0-6350-11e8-be86-73985bedf977
###### da489b20-8b52-11e7-b92d-d39e43e3de0f
###### dbb0ea90-4b12-11e9-94b3-a1bc5949ac31
###### Dionaea-ASN-Top-10
###### Dionaea-Countries-Top-10
###### Dionaea-Destination-Ports-Top-10
###### Dionaea-Events-by-Country-Histogram
###### Dionaea-Events-Histogram
###### Dionaea-Map
###### Dionaea-Password-Tagcloud-Large
###### Dionaea-Protocol
###### Dionaea-Source-IP-Top-10
###### Dionaea-Transport
###### Dionaea-Type
###### Dionaea-Username-Tagcloud-Large
###### e055e240-e851-11e8-97df-bbc3de28ece0
###### e18823d0-fa30-11e8-838f-fff066e21110
###### e1969e20-4878-11e8-9b3d-f36e8d4f5cb2
###### e3f00420-e7fd-11e8-9ac4-13ecd4ad8d70
###### e4b7cf40-8b52-11e7-b92d-d39e43e3de0f
###### e624bc50-7dd6-11e7-bee2-c98307c16efa
###### e77bc660-e9d3-11e8-be2c-8fd05c77f582
###### e7e43170-4723-11e9-8c08-4f3285877d8a
###### e9e534d0-6356-11e8-be86-73985bedf977
###### ea9b34a0-4b14-11e9-94b3-a1bc5949ac31
###### eaedcca0-fa2f-11e8-838f-fff066e21110
###### ec53e470-8376-11e7-9adb-2955c2136c8c
###### eca8e580-4877-11e8-9b3d-f36e8d4f5cb2
###### ef227eb0-7e9d-11e7-a286-9f03beba6417
###### ElasticPot-ASN-Top-10
###### ElasticPot-Countries-Top-10
###### ElasticPot-Events-by-Country-Histogram
###### ElasticPot-Events-Histogram
###### ElasticPot-Map
###### ElasticPot-Query-Top-10
###### ElasticPot-Source-IP-Top-10
###### f1a19000-7ebf-11e7-a286-9f03beba6417
###### f1f14c10-fa3a-11e8-838f-fff066e21110
###### f2742de0-e745-11e8-b4a6-215b0b97c069
###### f28b8c60-80e4-11e7-ba6f-4542711dd148
###### f4444100-e858-11e8-97df-bbc3de28ece0
###### f52ea410-3f57-11e9-8971-b9d98cc69287
###### f5e74220-e725-11e8-b4a6-215b0b97c069
###### f66b9200-e851-11e8-97df-bbc3de28ece0
###### f8e24f20-634e-11e8-be86-73985bedf977
###### fbf988c0-4b12-11e9-94b3-a1bc5949ac31
###### fcc64340-465f-11e9-9d82-ab3a5e06df4c
###### fcf68d60-fa34-11e8-838f-fff066e21110
###### fe02b580-4ad5-11e8-ab1b-fdef76c312f4
###### fe389160-4b14-11e9-94b3-a1bc5949ac31
###### ffb284f0-80cd-11e7-ab37-eb92b1bfb573
###### Glastopf-ASN-Top-10
###### Glastopf-Countries-Top-10
###### Glastopf-Events-by-Country-Histogram
###### Glastopf-Events-Histogram
###### Glastopf-Map
###### Glastop-Source-IP-Top-10
###### Honeytrap-ASN-Top-10
###### Honeytrap-Countries-Top-10
###### Honeytrap-Destination-Ports-Top-10
###### Honeytrap-Events-by-Country-Histogram
###### Honeytrap-Events-Histogram
###### Honeytrap-Map
###### Honeytrap-Source-IP-Top-10
###### NGINX-ASN-Top-10
###### NGINX-Countries-Top-10
###### NGINX-Events-by-Country-Histogram
###### NGINX-Events-Histogram
###### NGINX-HTTP-Method-Pie-Top-10
###### NGINX-HTTP-Status-Code-Pie-Top-10
###### NGINX-HTTP-User-Agent-Pie-Top-10
###### NGINX-Map
###### NGINX-Source-IP-Top-10
###### NGINX-Username-Tagcloud
###### P0f-OS-Top-10
###### Suricata-Alert-Signature-Top-10
###### Suricata-ASN-Top-10
###### Suricata-Countries-Top-10
###### Suricata-Events-by-Country-Histogram
###### Suricata-Events-Histogram
###### Suricata-Fileinfo-Magic-Top-10
###### Suricata-HTTP-Content-Type-Top-10
###### Suricata-HTTP-Hostname-Pie-Top-10
###### Suricata-HTTP-Method-Pie-Top-10
###### Suricata-HTTP-User-Agent-Pie-Top-10
###### Suricata-Map
###### Suricata-Source-IP-Top-10
###### Suricata-SSH-Client-Software-Version-Pie-Top-10

### Now importing 21 searches.
###### 2934abc0-4ad4-11e8-ab1b-fdef76c312f4
###### 3290fa70-69a2-11e7-bcac-d3ee6f9c26fd
###### 385ea460-ad22-11e8-942c-a39712fa9ddf
###### 557c7d60-f8ae-11e8-ad78-0555bc917463
###### 9c35dd90-6977-11e7-9c11-8d9c11943fa0
###### aa750980-8ab5-11e7-8fef-33e989079c7d
###### aea69a30-45d7-11e9-8f59-f1210ddd7670
###### baa53b00-b597-11e8-9a34-d951cebce834
###### c2bea500-47ca-11e8-a905-f74bbc7cbd2d
###### c3b89bc0-69a7-11e7-bcac-d3ee6f9c26fd
###### ConPot-Logs
###### Cowrie-Logs
###### d800f130-633f-11e8-be86-73985bedf977
###### Dionaea-Logs
###### ElasticPot-Logs
###### Glastopf-Logs
###### Honeypot-Logs
###### Honeytrap-Logs
###### NGINX-Logs
###### P0f-Logs
###### Suricata-Logs

### Statistics
###### Imported 943 index patterns.
###### Imported 19 dashboards.
###### Imported 255 visualizations.
###### Imported 21 searches.

[tsec@experimentalovercharge:~]$

By visiting :64297 I see the following messy page. Even after running systemctl restart tpot, I see the same messy page.

[root@experimentalovercharge:/home/tsec]# dps.sh
========| System |========
    Date:  Tue 09 Apr 2019 04:58:39 PM UTC
  Uptime:  16:58:39 up  7:30,  3 users,  load average: 4.55, 2.92, 1.52

NAME                  STATUS                       PORTS
adbhoney              Up 2 minutes                  0.0.0.0:5555->5555/tcp
ciscoasa              Up 2 minutes
conpot_guardian_ast   Up 2 minutes                  0.0.0.0:10001->10001/tcp
conpot_iec104         Up 2 minutes                  0.0.0.0:161->161/tcp, 0.0.0.0:2404->2404/tcp
conpot_ipmi           Up 2 minutes                  0.0.0.0:623->623/tcp
conpot_kamstrup_382   Up 2 minutes                  0.0.0.0:1025->1025/tcp, 0.0.0.0:50100->50100/tcp
cowrie                Up 2 minutes                  0.0.0.0:22-23->22-23/tcp
cyberchef             Up 2 minutes (healthy)        127.0.0.1:64299->8000/tcp
dionaea               Up 2 minutes
elasticpot            Up 2 minutes                  0.0.0.0:9200->9200/tcp
elasticsearch         Up 2 minutes (healthy)        127.0.0.1:64298->9200/tcp
ewsposter             Up 2 minutes
head                  Up About a minute (healthy)   127.0.0.1:64302->9100/tcp
heralding             Up 2 minutes                  0.0.0.0:110->110/tcp, 0.0.0.0:143->143/tcp, 0.0.0.0:993->993/tcp, 0.0.0.0:995->995/tcp, 0.0.0.0:1080->1080/tcp, 0.0.0.0:5432->5432/tcp, 0.0.0.0:5900->5900/tcp
honeytrap             Up 2 minutes
kibana                Up About a minute (healthy)   127.0.0.1:64296->5601/tcp
logstash              Up About a minute (healthy)
mailoney              Up 2 minutes                  0.0.0.0:25->25/tcp
medpot                Up 2 minutes                  0.0.0.0:2575->2575/tcp
nginx                 Up 2 minutes
p0f                   Up 2 minutes
rdpy                  Up 2 minutes                  0.0.0.0:3389->3389/tcp
snare                 Up 2 minutes                  0.0.0.0:80->80/tcp
spiderfoot            Up 2 minutes (healthy)        127.0.0.1:64303->8080/tcp
suricata              Up 2 minutes
tanner                Up 2 minutes
tanner_api            Up 2 minutes
tanner_phpox          Up 2 minutes
tanner_redis          Up 2 minutes                  6379/tcp
tanner_web            Up 2 minutes
t3chn0m4g3 commented 5 years ago

This is expected. You just need to define the pattern, i.e. logstash-* and based on @timestamp.

mahmoodn commented 5 years ago

OK. I selected logstash-* and I see https://pasteboard.co/I9sPZKZ.png I didn't get the point about timestamp. Can you explain more? Moreover, the default date of the new installation is 

[tsec@experimentalovercharge:~]$ date
Wed 10 Apr 2019 08:58:21 AM UTC

The time is wrong based on my local time. It should be +4:30. Does that matter?

t3chn0m4g3 commented 5 years ago

1) Please understand this goes beyond the scope of what can be explained in an issue. Please check the Kibana documentation. 2) Timezone has to be UTC.

mahmoodn commented 5 years ago

1- That is a big documentation. I actually don't know which section is related to my problem. Upgrading? I am guessing if I ask the question on Kibana's list, they say it is a problem with tpot. Which version of Kibana is used in tpot? I didn't see that in release note.

2- So, when my local time is 13:30 and tpot time is 9:00 and I look at "last 15 minutes", does it point to 13:15 ~ 13:30 or 8:45 ~ 9:00?