The header content contains invalid characters

[ne0c0de]

Hi,

We're using logdna-agent version 2.1.2-1 on our server.

It's stopped to sending logs to logdna server and giving this error massively:

2021-01-14 08:50:55 0000 [error] { Error: A connection-based error occurred that will not be retried. See meta data for details.
    at process.nextTick (/usr/bin/node_modules/@logdna/logger/lib/logger.js:771:25)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)
  meta:
   { actual: 'The header content contains invalid characters',
     code: undefined,
     firstLine: 'log line text',
     lastLine: '\t',
     retrying: false,
     attempts: 1 } }
2021-01-14 08:50:58 0000 [error] { Error: A connection-based error occurred that will not be retried. See meta data for details.
    at process.nextTick (/usr/bin/node_modules/@logdna/logger/lib/logger.js:771:25)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)

I tried to update the logdna agent version but nothing change.

How can I resolve this problem?

[darinspivey]

This error is typically accurate, meaning that the headers have invalid characters somewhere in them.

• Can you make sure that your "logdna key" is set correctly (no newlines, etc)? The authorization header is created from this.
• What is the hostname of the machine running the agent?
• Have you modified any of the URLs the agent uses for ingestion, or is it just the default? Sometimes, invalid URL characters can cause this error
• Lastly, I'll ask if this problem appeared without changing anything, or had something changed or been upgraded prior to seeing the error?

Based on what you've sent, these version are stable and have not changed for some time which leads us to believe that the error truly is in the value of the headers somewhere. Debugging this will be tricky since the logger is a dependency of the agent, but we can perhaps guide you on adding some debug statements to expose some of the values (if that's possible for you to do).

[ne0c0de]

Hi @darinspivey

Thanks for your answer.

Nothing change on the configuration that was working before. All the configuration that is working on other servers only hostname different with few characters

I'm also an nodejs developer, so i would like to make debug with your guidelines

[darinspivey]

Ok, so no immediate changes—that's good. The quickest way to gather some more information is just to see the headers themselves. If you're able to modify your node_modules/@logdna/logger/lib/logger.js file, then let's dump the headers as part of the error message. You should be on version 2.0.0 of the logger, so on this line:

https://github.com/logdna/logger-node/blob/21b07593a25bb9a686879fbbb29c7509ba772c6e/lib/logger.js#L733

Add this as part of the errorMeta object.

, headers: config.headers

If the error comes out too nested such that the values are not clearly visible, you can always JSON.stringify(config.headers). We'll be looking at things like user-agent which can sometimes have a lot of information in it.

Obviously, if we can know what it's failing, we'll know how to proceed with a fix on either server side or consumer side. Just please be careful not to post sensitive values like your authorization header on this post. Sorry, I had to say it :)

[ne0c0de]

it seems like logdna is compiled binary in our server (Centos7) so I couldn't find any node_modules fodler neither logdna. Should I compile it from source with editing it?

[ne0c0de]

@darinspivey

[darinspivey]

Ah, yes, that's important to clarify. From here on out, I'll assume that you have access to start/stop services on the server and keep it up while you manually play with things. If you have at least node 10 on the box, then you can just install the node package and run it directly as root. The binary build just encapsulates the node runtime and includes some init.d scripts. If you're able to run it directly, and keep it running while the server is up, then this should work.

/etc/init.d/logdna-agent stop
cd
mkdir agenttest && cd agenttest
npm install logdna@latest --no-save
node index.js

See if that yields the same results. In the meantime, we need to bump @logdna/logger to be current. Although I don't think your issue would be fixed by that bump, it's a golden rule to be current before debugging anything. We will start that upgrade, and in the meantime, you can do the manual testing to not be blocked by our efforts.

[ne0c0de]

I have root user on server but node didn't installed on it and it's little bit dangrous to install because it's production database master server. is there any way to debug it?

[darinspivey]

Hmm, ok. We can get this fixed, it'll just require a few code changes, so bear with us. Our agent needs to have its @logdna/logger bumped anyway. I think now is the time to add some debugging features to the logger that can be activated from the agent. We can accomplish that in 1 release, pending any logger changes. At a minimum, the error message should contain more context like the headers.

There's plenty of things you might be able to do from the server, but none that I can walk you through (HTTP traffic interception to see headers, etc). We will get right on these upgrades for you. Sorry there isn't a quicker answer.

[darinspivey]

Keeping this open, as the new PR won't directly solve your issue. Once we upgrade the agent, you can bump your version and we'll see if we can't get some more context for your error.

[ne0c0de]

so can I upgrade the agent now to see more debug messages?

[jorgebay]

The agent release is still pending, we will keep you posted.

[darinspivey]

@ne0c0de a new version of the agent should be available for linux now (Windows probably tomorrow). Assuming you're on linux, see if you can bump it and get some more context on what your error might be. What are your headers?

[ne0c0de]

Hello @darinspivey here's the detail log:

{ Error: A connection-based error occurred that will not be retried. See meta data for details.
    at process.nextTick (/usr/bin/node_modules/@logdna/logger/lib/logger.js:777:25)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)
  meta:
   { actual: 'The header content contains invalid characters',
     code: undefined,
     firstLine: 'xxxxx',
     lastLine: 'zzzz',
     retrying: false,
     attempts: 1,
     headers:
      { 'Content-Type': 'application/json; charset=UTF-8',
        'Content-Encoding': 'gzip',
        'user-agent': '@logdna/logger/2.2.1 (logdna-agent/2.2.0 CentOS/8.3.2011\n)' },
     url: 'https://logs.eu-de.logging.cloud.ibm.com:443/logs/agent?now=1611748044068&hostname=xxx-yyy.zzz.cloud&mac=00%3A00%3A00%3A00%3A00%3A00&ip=11.111.11.99&tags=xxx-db1' } }

hostname, ip and tags values are changed here but there's a normal characters in real lines.

[evanlucas]

Ah there is a new line in the user-agent header @darinspivey

[ne0c0de]

I guess it's reading that data from somewhere in the system. can I have a chance to manipulate it? @evanlucas ?

[darinspivey]

Ah, great! That's the info we needed. Yes, the newline is the problem. This is being sent over from our agent, so we will take a look there. The fix should be straight forward (hopefully I'm not jinxing it ;)

Thanks for your patience while we work through this.

[darinspivey]

I see no reason why the fix can't be in the logger. It should remove invalid characters no matter where they come from. We will be able to fix it quicker this way. Standby.

[ne0c0de]

hello @darinspivey is that commit released officially or I should wait for some time for it?

[jorgebay]

It's available in the npm registry, version 2.2.2: https://www.npmjs.com/package/@logdna/logger

[ne0c0de]

I think it's not ready yet in Centos Yum repo right? Because whenever I try to update it's saying that Package logdna-agent-2.2.0-1.x86_64 is already installed

[jorgebay]

oh, I misunderstood the question, sorry.

Indeed, the agent hasn't been released yet.

[darinspivey]

Sorry, @ne0c0de , the issue was auto-closed after merge. The logger is a dependency of the agent, so we have to do another release of that before you'll see the true fix. Sorry for the delay.

[smusali]

LogDNA Agent v2.2.1 which contains @logdna/logger@2.2.2 has just been released for Linux.

[darinspivey]

@ne0c0de Please confirm that your issue is resolved, thanks!

[ne0c0de]

Yes, I just updated agent today and it working without any error. Thank you for your help