TCK log file results and results reporting

Mahersrasra commented 1 year ago

What do you want to know?

Hi ,

I'm in the process of ensuring my MQTT broker meets the Sparkplug compliance standards. While conducting tests, the results were recorded in the web console, but I couldn't find any indication of a corresponding log file or any related information. I'm looking to locate the potential log file, if it exists, and determine whether there's an underlying problem that needs to be addressed.
Additionally, to obtain the certification, I understand that a report results file needs to be generated. I would appreciate guidance on the procedure for creating this report file.

Is this related to a Sparkplug Listing request? If so, link the issue from https://github.com/eclipse-sparkplug/sparkplug.listings here.

No response

Version

3.0.0 (Default)

Accept EFTL Terms

[X] I agree to the terms of EFTL

bryce-nakatani commented 1 year ago

For the TCK, you are running an instance of HiveMQ Community. In the HiveMQ Community ./bin directory, there should be a file named SparkplugTCKResults.log. This contains your test results.

The log must be post processed through a python script. Copy the log file into the ./SparkplugTCK directory . Then you'll run the python script, something like "python3 report.py SparkplugTCKResults.log". The processed result is an HTML file in the same directory .

But before you can process the log, in the 3.0.0 version of the TCK, there is a patch you need to manually perform. Download this TCK source tag https://github.com/eclipse-sparkplug/sparkplug/tree/v3.0.0. You'll need to recursively copy the directory ./sparkplug-3.0.0/tck/src to ./SparkplugTCK/src. The python script requires the source files for reference when post processing. It was accidentally left out of the 3.0.0 release. This is something we will patch up in a future release.

Review your HTML report and follow the instructions https://github.com/eclipse-sparkplug/sparkplug.listings to "Get Listed". We look forward to seeing your request.

On Thu, Aug 17, 2023 at 8:35 AM Mahersrasra @.***> wrote:

What do you want to know?

Hi ,

-

I'm in the process of ensuring my MQTT broker meets the Sparkplug compliance standards. While conducting tests, the results were recorded in the web console, but I couldn't find any indication of a corresponding log file or any related information. I'm looking to locate the potential log file, if it exists, and determine whether there's an underlying problem that needs to be addressed.

Additionally, to obtain the certification, I understand that a report results file needs to be generated. I would appreciate guidance on the procedure for creating this report file.

Is this related to a Sparkplug Listing request? If so, link the issue from https://github.com/eclipse-sparkplug/sparkplug.listings here.

No response Version

3.0.0 (Default) Accept EFTL Terms

I agree to the terms of EFTL

— Reply to this email directly, view it on GitHub https://github.com/eclipse-sparkplug/sparkplug/issues/476, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEF6AZVKYLIJFVQSKC7YPQTXVY23LANCNFSM6AAAAAA3UINZVU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Mahersrasra commented 1 year ago

Hello,

I wanted to thank you for your quick and informative response regarding the outlined steps. Nevertheless, I'm still facing an ongoing concern pertaining to the generation of logs. As you indicated, I couldn't find any log files within the hive mq bin directory. I would greatly appreciate any additional advice or suggestions you could offer to help address this particular issue. as you can see below the web console is logging the results

but under the hivemq bin directory there is no footprint of a log file

bryce-nakatani commented 1 year ago

I use Ubuntu Server to run the TCK and HiveMQ. When I start HiveMQ, I initiate run.sh from the context of the bin folder.

I just ran the TCK's test for Mosquitto 2.0.15. It produced a PASS for the Sparkplug Compliant test. The SparkplugTCKResults.log was generated in the /hivemq-ce.../bin folder. It's contents are:

@.***:~/hivemq-ce-2022.1/bin$ cat SparkplugTCKResults.log 2023-08-18 07:49:53.354 Summary Test Results for Broker Sparkplug Compliant conformance-mqtt-qos0: PASS; conformance-mqtt-qos1: PASS; conformance-mqtt-retained: PASS; conformance-mqtt-will-messages: PASS; OVERALL: PASS;

I'm not familiar with how the Windows OS affects where the results are being stored and it depends how you instantiate the application. Maybe search specific drives for where the file might be stored.

On Fri, Aug 18, 2023 at 2:18 AM Mahersrasra @.***> wrote:

Hello,

I wanted to thank you for your quick and informative response regarding the outlined steps. Nevertheless, I'm still facing an ongoing concern pertaining to the generation of logs. As you indicated, I couldn't find any log files within the hive mq bin directory. I would greatly appreciate any additional advice or suggestions you could offer to help address this particular issue. as you can see below the web console is logging the results

[image: image] https://user-images.githubusercontent.com/78274224/261541202-0c11de69-5572-4e94-90ab-758b41315bb2.png

but under the hivemq bin directory there is no footprint of a log file

[image: image] https://user-images.githubusercontent.com/78274224/261560776-992825ae-2e99-4f34-a1a3-1e986ee80624.png

— Reply to this email directly, view it on GitHub https://github.com/eclipse-sparkplug/sparkplug/issues/476#issuecomment-1683617735, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEF6AZTODDDFDRGPA6CYAATXV4XN7ANCNFSM6AAAAAA3UINZVU . You are receiving this because you commented.Message ID: @.***>

Mahersrasra commented 1 year ago

Thank you! I'll look into this and get back to you once I have more information.

Mahersrasra commented 1 year ago

Hello once again,

Apologies for any inconvenience. It seems that the steps you followed didn't yield the expected results. I suspect that the issue might be connected to the way log generation is handled. As evidenced below during the startup of the broker, a folder is created containing an "lck" file, but there is no indication of log traces.

I would greatly appreciate any assistance you could provide regarding this situation.

icraggs-sparkplug commented 1 year ago

I used HiveMQ community edition and the log results file was written to the bin directory:

 Directory of C:\Users\Ian\hivemq-ce-2023.6\bin

25/08/2023  17:55    <DIR>          .
25/08/2023  17:55    <DIR>          ..
25/08/2023  14:54             4,430 diagnostics.bat
25/08/2023  14:54             4,545 diagnostics.sh
25/08/2023  14:54        99,841,838 hivemq.jar
25/08/2023  14:54    <DIR>          init-script
25/08/2023  14:54             4,451 run.bat
25/08/2023  14:54             4,557 run.sh
25/08/2023  17:56             2,384 SparkplugTCKResults.log
25/08/2023  17:55    <DIR>          SparkplugTCKResultsCollector-tcplocalhost1883
25/08/2023  14:54               688 windows-service.txt

The HiveMQ console looked like this on startup:

2023-08-25 17:55:49,098 INFO  - Starting HiveMQ Community Edition Server
2023-08-25 17:55:49,098 INFO  - HiveMQ version: 2023.6
2023-08-25 17:55:49,098 INFO  - HiveMQ home directory: C:\Users\Ian\hivemq-ce-2023.6
2023-08-25 17:55:49,100 INFO  - Log Configuration was overridden by C:\Users\Ian\hivemq-ce-2023.6\conf\logback.xml
2023-08-25 17:55:49,256 INFO  - This HiveMQ ID is VN4Ud
2023-08-25 17:55:50,893 INFO  - Starting with file persistence mode.
2023-08-25 17:55:51,214 INFO  - Starting HiveMQ extension system.
2023-08-25 17:55:51,254 WARN  -
################################################################################################################
# This HiveMQ deployment is not secure! You are lacking Authentication and Authorization.                      #
# Right now any MQTT client can connect to the broker with a full set of permissions.                          #
# For production usage, add an appropriate security extension and remove the hivemq-allow-all extension.       #
# You can download security extensions from the HiveMQ Marketplace (https://www.hivemq.com/extensions/).       #
################################################################################################################
2023-08-25 17:55:51,255 INFO  - Extension "Allow All Extension" version 1.0.0 started successfully.
2023-08-25 17:55:51,257 INFO  - Starting Sparkplug TCK Extension
2023-08-25 17:55:51,262 INFO  - Sparkplug TCK message monitor 1.0
2023-08-25 17:55:51,263 INFO  - Sparkplug TCK Host Application utility starting
2023-08-25 17:55:51,265 INFO  - Sparkplug TCK Edge Node utility
2023-08-25 17:55:51,266 INFO  - Sparkplug TCK Results Collector initializing
2023-08-25 17:55:51,272 INFO  - Extension "EclipseÔäó SparkplugÔäó TCK" version 3.0.0 started successfully.
2023-08-25 17:55:51,291 INFO  - Starting TCP listener on address 0.0.0.0 and port 1883
2023-08-25 17:55:51,385 INFO  - Starting Websocket listener on address 0.0.0.0 and port 8000
2023-08-25 17:55:51,388 INFO  - Started TCP Listener on address 0.0.0.0 and on port 1883
2023-08-25 17:55:51,388 INFO  - Started Websocket Listener on address 0.0.0.0 and on port 8000
2023-08-25 17:55:51,390 INFO  - Started HiveMQ in 2295ms
2023-08-25 17:55:52,631 INFO  - Sparkplug TCK Results Collector: Setting new result log file: SparkplugTCKResults.log
2023-08-25 17:55:52,632 INFO  - Sparkplug TCK Results Collector: New log file created: C:\Users\Ian\hivemq-ce-2023.6\bin\SparkplugTCKResults.log
2023-08-25 17:55:52,633 INFO  - Sparkplug TCK Results Collector: setting UTCwindow to 60000

It would be useful to see what messages there are on your HiveMQ console to compare. The "log file created" message is a DEBUG level message - you can show that by changing your HiveMQ log level to DEBUG by changing the entry in the HiveMQ logback.xml file in the conf directory from:

    <root level="${HIVEMQ_LOG_LEVEL:-INFO}">
        <appender-ref ref="FILE"/>
        <appender-ref ref="CONSOLE"/>
    </root>

to:

    <root level="${HIVEMQ_LOG_LEVEL:-DEBUG}">
        <appender-ref ref="FILE"/>
        <appender-ref ref="CONSOLE"/>
    </root>

Mahersrasra commented 1 year ago

Hello , I adjusted the log level as you suggested, and I've retrieved the following information from the HiveMQ console: "Can't open directory channel." I did some research but I'm still uncertain about the cause of this message.

Mahersrasra commented 1 year ago

Closing this issue since it appears that the logs generation was hindered by security policies on my system.

icraggs-sparkplug commented 1 year ago

Thanks for the info

eclipse-sparkplug / sparkplug