Unexpected character/Authentication failed when running

[stuartluscombe]

I'm just testing out Zabbix and have compiled zabbix-map and rebuilt lldp_get.so. When running it I'm getting the errors 'unexpected character' and 'Authentication failed'.

This is with the current version of Zabbix (5.4.7). I wonder if this is to do with the changes in API access?

./zabbix-map -map "test" -ip "10.1.100.0/24" -u Admin -p zabbix
Error: unexpected character
Authentication failed for user Admin accessing Zabbix

[TiggyWiggler]

I have not tried running the application against the new latest update but I do not see anything in the documentation that is different to what I was doing before. there is nothing here that seems out of order, but I will need to test to know for sure. I am going to make a 5.4.7 dev build now and I will report back.

[stuartluscombe]

I've resolved the authentication part of the issue, I needed to enter the FQDN as the endpoint (it didn't like localhost).

After this I was able to create a map with the few devices we added in, however as we've added more (all HP/Aruba switches) I'm now getting the 'unexpected character' error again but I don't know which device is being processed at the time the error occurs.

[TiggyWiggler]

Thank you for this. When the program downloads the host list from the Zabbix server it saves it as a cache file. This file can then be shared so that we can step through the process in our code and see where the error is thrown. What I would suggest in the first instance is that you try to run the code against the cache file on your local machine and see if the 'unexecpted character' error occurs using the cache file and if it does please share the file with us and we can check it here. I'm a bit busy (only got the 5.4.7 build running last night before EOB) but I will try to do what I can later today. So looking at your original post you would want to do it this way. First run:

./zabbix-map -map "test" -ip "10.1.100.0/24" -u Admin -p zabbix -cache "myfile"

This will create a cache file called "myfile" you will likely get your error again. Next you want to run against the cache file to see if the error is reproduced.

./zabbix-map -map "test" -ip "10.1.100.0/24" -u Admin -p zabbix -cache "myfile" -src file

If you get the error here then share the file myfile with me and I can look from this side.

As a thought, you may want to see what happens if you turn off pseudo objects as shown below, it may stop the problem altogether:

./zabbix-map -map "test" -ip "10.1.100.0/24" -u Admin -p zabbix -phosts 0 -phubs 0

good luck.

[stuartluscombe]

I tried out the different options you suggested.

I did get the same error when creating a cachefile and when checking the file the only contents is the word 'null'.

Running the command to turn off pseudo objects made no difference, I still received the same error.

[TiggyWiggler]

I have released a new version with more debug tracing. Can you run again with the switch '-debug true' and tell me where the unexpected character bit appears please? you will need to download the latest code and compile.

[stuartluscombe]

Thanks for helping with this. Here are the debug results:

$  ./zabbix-map -ep http://zabbix01.mydomain.local/api_jsonrpc.php -map "test" -ip "10.1.100.0/24" -u Admin -p zabbix -debug true
DEBUG: Debug Enabled
API End Point: http://zabbix01.mydomain.local/api_jsonrpc.php
Map Name: test
IP: 10.1.100.0/24
Source: api
Cache:
Order By: 1
Padding: 50.0, 50.0, 50.0, 50.0
Node Spacing: 100.0,100.0
Output: api
Username: Admin
Password: zabbix
LinkLabels: 0
Pseudo Hubs: 1
Pseudo Hosts: 1
DEBUG: About to authenticate
DEBUG: zconnResp [user.login]
DEBUG: Authentication complete
DEBUG: parseSpacing.
DEBUG: parsePadding
DEBUG: parseSorts
DEBUG: zconnGetHostsFromAPI
DEBUG: zconnResp [host.get]
Error: unexpected character
DEBUG: About to check IP addresses
DEBUG: About to evaluate hosts
DEBUG: freeHostCol

[TiggyWiggler]

Thank you for this. I have added additional tracing at the point that is indicated above so please download and compile again and share the trace output. If the problem is occurring in the parser then we will see the 'unexpected character' immediately after the debug line "DEBUG: zconnParseHosts" and I really do not want it to be in that area!

[stuartluscombe]

Here is the latest output:

$ ./zabbix-map -ep http://zabbix01.mydomain.local/api_jsonrpc.php -map "test" -ip "10.1.100.0/24" -u Admin -p zabbiix -debug true
DEBUG: Debug Enabled
API End Point: http://zabbix01.mydomain.local/api_jsonrpc.php
Map Name: test
IP: 10.1.100.0/24
Source: api
Cache:
Order By: 1
Padding: 50.0, 50.0, 50.0, 50.0
Node Spacing: 100.0,100.0
Output: api
Username: Admin
Password: zabbix
LinkLabels: 0
Pseudo Hubs: 1
Pseudo Hosts: 1
DEBUG: About to authenticate
DEBUG: zconnResp [user.login]
DEBUG: zconnResp.curl_easy_perform []
DEBUG: zconnResp.curl_easy_perform complete
DEBUG: zconnResp.curl_easy_perform CURLE_OK
DEBUG: parseResult
DEBUG: Authentication complete
DEBUG: parseSpacing.
DEBUG: parsePadding
DEBUG: parseSorts
DEBUG: zconnGetHostsFromAPI
DEBUG: zconnResp [host.get]
DEBUG: zconnResp.curl_easy_perform []
DEBUG: zconnResp.curl_easy_perform complete
DEBUG: zconnResp.curl_easy_perform CURLE_OK
Error: unexpected character
DEBUG: About to check IP addresses
DEBUG: About to evaluate hosts
DEBUG: freeHostCol

[TiggyWiggler]

okay. so the problem is occurring literally just before the cache file is created, which is why it is null when you try to open it. I have now added an additional trace command into the debug but it creates a very large output and it will contain sensitive information. For this purpose I recommend to output to a file. e.g. ./zabbix-map -debug true > output.txt and then email it directly to me instead of pasting here. craig@deadline-automation.com. I will take a look at the file and see if the cause is clear to me.

[stuartluscombe]

I didn't get the expected output (as in your demo.txt), there's nothing sensitive here so I'll paste it below:

DEBUG: Debug Enabled
API End Point: http://zabbix01.mydomain.local/api_jsonrpc.php
Map Name: test
IP: 10.1.100.0/24
Source: api
Cache:
Order By: 1
Padding: 50.0, 50.0, 50.0, 50.0
Node Spacing: 100.0,100.0
Output: api
Username: Admin
Password: zabbix
LinkLabels: 0
Pseudo Hubs: 1
Pseudo Hosts: 1
DEBUG: About to authenticate
DEBUG: zconnResp [user.login]
DEBUG: zconnResp.curl_easy_perform
DEBUG: zconnResp.curl_easy_perform complete
DEBUG: zconnResp.curl_easy_perform CURLE_OK
DEBUG: curl response: {"jsonrpc":"2.0","result":"d2350d6b94e395214a10d4a69b874e26","id":"1"}
DEBUG: parseResult
DEBUG: Authentication complete
DEBUG: parseSpacing.
DEBUG: parsePadding
DEBUG: parseSorts
DEBUG: zconnGetHostsFromAPI
DEBUG: zconnResp [host.get]
DEBUG: zconnResp.curl_easy_perform
DEBUG: zconnResp.curl_easy_perform complete
DEBUG: zconnResp.curl_easy_perform CURLE_OK
DEBUG: curl response: `<CA>4<8D><C8>U
DEBUG: About to check IP addresses
DEBUG: About to evaluate hosts
DEBUG: freeHostCol

The characters in the curl response look like a MAC address?

[TiggyWiggler]

Thank you for staying with me through this entire debug process. I am sure your time is valuable and I really appreciate your support in trying to refine this community project.

Looking at your post, the entire format of that response is wrong. When Zabbix responds to the request it should always start "{"jsonrpc":"2.0","result":" because it is a JSON response from the server. below is a snippet of what I get from my server:

{"jsonrpc":"2.0","result":[{"hostid":"10535","host":"10.16.161.53","items":[{"itemid":"62794","name":"ICMP ping","value_type":"3","lastvalue":"0"},

Even if the response was empty it would still start with the jsonrpc header. Something is going wrong in the way that the response is being read or decoded, maybe from a null terminator in a place that it is not expected. I have not seen this before. I will do some reading into the API but for now I am at the limit of my knowledge.

edit: I just noticed that you did not get your error on the last run either. Interesting

[TiggyWiggler]

I have made a minor change to how the debug is output. Can you try this please? If this doesn't work then I am stuffed.

[stuartluscombe]

I appreciate all of the help with this. I think I'm going to have to roll my sleeves up on this one as it's not doing what you're expecting. As it did initially work I feel like it's something in our database that's causing the issue.

DEBUG: Debug Enabled
API End Point: http://zabbix01.mydomain.local/api_jsonrpc.php
Map Name: testing
IP: 10.1.100.0/24
Source: api
Cache:
Order By: 1
Padding: 50.0, 50.0, 50.0, 50.0
Node Spacing: 100.0,100.0
Output: api
Username: Admin
Password: zabbix
LinkLabels: 0
Pseudo Hubs: 1
Pseudo Hosts: 1
DEBUG: About to authenticate
DEBUG: zconnResp [user.login]
DEBUG: zconnResp.curl_easy_perform
DEBUG: zconnResp.curl_easy_perform complete
DEBUG: zconnResp.curl_easy_perform CURLE_OK
DEBUG: curl response size: 70, curl response: {"jsonrpc":"2.0","result":"a492ef0a60826e7d2795ecdec3fb7ab5","id":"1"}
DEBUG: parseResult
DEBUG: Authentication complete
DEBUG: parseSpacing.
DEBUG: parsePadding
DEBUG: parseSorts
DEBUG: zconnGetHostsFromAPI
DEBUG: zconnResp [host.get]
DEBUG: zconnResp.curl_easy_perform
DEBUG: zconnResp.curl_easy_perform complete
DEBUG: zconnResp.curl_easy_perform CURLE_OK
DEBUG: curl response size: 0, curl response: `��)V
Error: unexpected character
DEBUG: About to check IP addresses
DEBUG: About to evaluate hosts
DEBUG: freeHostCol

[stuartluscombe]

I've spent the morning doing some digging about. My knowledge of C/C++ is non-existent, but I have experience in other languages so I found my way around.

I managed to re-create the issue (get an API error) using PowerShell. The problem is occurring when collecting data from the 'selectItems' part of the host.get query. If that isn't included then the error doesn't occur.

After replicating the error I thought there must be an error on the Zabbix server somewhere and checking in the Nginx logs I found the culprit

"PHP message: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /usr/share/zabbix/include/classes/api/managers/CHistoryManager.php on line 280"

I increased the memory limit in /etc/zabbix/php-fpm.conf to 256M, and then 512M when that didn't work, restarted the zabbix-server and php7.4-fpm services and now the map creation is working again!

It looks like Zabbix was having an issue when trying to sort the number of items given the point at which the error occurred

CArrayHelper::sort($values, [
    ['field' => 'clock', 'order' => ZBX_SORT_DOWN],
    ['field' => 'ns', 'order' => ZBX_SORT_DOWN]
]);

[TiggyWiggler]

Well done for finding that! That is really interesting and not something I have seen before. It makes me wonder why the Zabbix API does not respond with an error message but instead fails internally and then returns rubbish to the requester. I might have to review the parser to ensure that I am reading out error codes correctly. I have tested this code on thousands of hosts without a problem but you seem to be hitting limitations of the API so I can only assume that you are getting loads of items per host or lots of hosts.

Are you happy for me to close this ticket?