The deployment of the Diskoverdata application is not working

[sync-by-unito[bot]]

1. Install the Diskoverdata application
2. Expected result: the application will start
3. Actual result:

• the log of the 'diskoverdata' container is displaying an ElasticSearch error and stack trace
• whey trying to see the logs of the elasticsearch app, I get a 400 error

Host ID: 6aff5690762bff7666fa50681dd63d32ddb3901d8a5c8e2e8feff810672ca51a

Session ID: 3cd0f43a-30c2-6f03-05a5-3f51c299a9d8

┆Attachments: debug-TrueNAS-20240620094924.tgz | Screen Shot 2024-06-20 at 9.47.58 AM.png | Screen Shot 2024-06-20 at 9.48.34 AM.png | Screen Shot 2024-06-21 at 9.03.24 AM.png

[sync-by-unito[bot]]

➤ Bug Clerk commented:

Thank you for submitting this TrueNAS Bug Report! So that we can quickly investigate your issue, please attach a Debug file and any other information related to this issue through our secure and private upload service below. Debug files can be generated in the UI by navigating to System > Advanced > Save Debug.

https://ixsystems.atlassian.net/servicedesk/customer/portal/15/group/37/create/153

[sync-by-unito[bot]]

➤ Bogdan Solga commented:

The stack trace from the diskoverdata container of the diskoverdata-elasticsearch-598b655fc7-8x95s Pod:

Created elasticsearch keystore in /usr/share/elasticsearch/config

2024-06-20 10:19:33.487234+03:00OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. 2024-06-20 10:19:35.015797+03:00{"type": "server", "timestamp": "2024-06-20T10:19:35,007+03:00", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "diskoverdata", "message": "uncaught exception in thread [main]", 2024-06-20 10:19:35.015883+03:00"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];", 2024-06-20 10:19:35.015894+03:00"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.015902+03:00"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.015930+03:00"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.015938+03:00"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.015945+03:00"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.015956+03:00"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.015963+03:00"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.015971+03:00"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service", 2024-06-20 10:19:35.015978+03:00"at org.elasticsearch.node.Node.(Node.java:607) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.015989+03:00"at org.elasticsearch.node.Node.(Node.java:253) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.015996+03:00"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:241) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016003+03:00"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016014+03:00"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:369) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016022+03:00"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016031+03:00"... 6 more", 2024-06-20 10:19:35.016039+03:00"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes", 2024-06-20 10:19:35.016050+03:00"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]", 2024-06-20 10:19:35.016057+03:00"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]", 2024-06-20 10:19:35.016065+03:00"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]", 2024-06-20 10:19:35.016073+03:00"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]", 2024-06-20 10:19:35.016086+03:00"at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]", 2024-06-20 10:19:35.016096+03:00"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]", 2024-06-20 10:19:35.016104+03:00"at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]", 2024-06-20 10:19:35.016125+03:00"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016151+03:00"at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:209) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016162+03:00"at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:269) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016170+03:00"at org.elasticsearch.node.Node.(Node.java:273) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016177+03:00"at org.elasticsearch.node.Node.(Node.java:253) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016189+03:00"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:241) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016196+03:00"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016203+03:00"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:369) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016214+03:00"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.2.jar:7.5.2]", 2024-06-20 10:19:35.016224+03:00"... 6 more"] }

[sync-by-unito[bot]]

➤ Bogdan Solga commented:

The error displayed when trying to get the logs of the diskoverdata container from the diskoverdata-746999d48d-2cxw7 Pod:

Received 400 response code from '/api/v1/namespaces/ix-diskoverdata/pods/diskoverdata-746999d48d-2cxw7/log?follow=true×tamps=true&timeoutSeconds=1800&container=diskoverdata&tailLines=500'

[sync-by-unito[bot]]

➤ Daniel Pizappi commented:

Thanks for your submission! This is in our queue to review now. An engineering representative will update with any further questions or details in the near future.

[sync-by-unito[bot]]

➤ Stavros Kois commented:

Hello, if you used host paths for data, ensure that the user and group IDs you defined in the app wizard have read/write access. The logs indicate that there is an issue with permissions

[sync-by-unito[bot]]

➤ Bogdan Solga commented:

Hello, Stavros Kois!

I have just checked (again), and the user that I have configured in the app wizard is the owner of the path that is mounted in the container. I have tried with a different ZFS dataset, and the problem is the same.

Given that all the app settings are in different ixVolumes (see the attached screenshot), why would there be a problem with the permissions for an user mounted storage? That might also be read only, if needed

[sync-by-unito[bot]]

➤ Bogdan Solga commented:

Stavros Kois - for debugging purposes, I have installed the app without any host path mounted, and the behavior is exactly the same. Moreover, the Java stack trace clearly shows that the issue is in the container (AccessDeniedException: /usr/share/elasticsearch/data/nodes), not in any user mounted host/SMB path

[sync-by-unito[bot]]

➤ Stavros Kois commented:

Sorry, was looking at Elastic Search standalone app’s code instead of diskoverdata’s elastic search. Found the issue.

Thanks

[sync-by-unito[bot]]

➤ Bogdan Solga commented:

Thank you, Stavros Kois 🙂 Glad you found the issue, looking forward for the updated chart