Ideolog not parsing log but freezes

[a-and-m]

Hi,

I am using PHPStorm and I would like to look at some log files with it. The log-files are created using node and log4js. I entered a custom format which I think is correct, but it is not working. The log entries look like this:

[2019-04-05T00:00:00.020] [INFO] job.sendShoppingCartDropOutMails - ---> Starting job sendShoppingCartDropOutMails
[2019-04-05T00:00:00.025] [INFO] job.processMailSequences - ---> Starting job processMailSequences
[2019-04-05T00:00:00.025] [INFO] job.processMailSequences - Sending all mails in sequences that must be sent by now

I have entered the following message pattern: ^\[([^\s]+)\]\s\[(\w+)\]\s([\w\.]+)\s-\s(.*)$

Message start pattern: ^\[

Time format: yyyy-MM-ddTHH:mm:ss.SSS

I think these settings are correct, but when I try to open such a log-file, it does not work, and starts to behave strange, I also had freezes.

Is there something wrong with these settings?

Thanks for your help!

[knah]

I'm currently working on update for the plugin that should greatly improve performance and get rid of freezes. I also hope to include some usability improvements, such as regex validation and previewing in log format settings. I don't really see what is wrong with your specific pattern. You may want to disable other log formats and reopen your log file to see if it detects the wrong format.

[a-and-m]

Hi,

disabling the other formats does not help, it's still freezing. If I disable my own pattern I can open the log files (but without highlighting). So it's definitely related to that custom format.

[knah]

Please try Ideolog 0.9.0, it contains significant improvements for performance and fixes numerous reasons for freezes. Let me know if the issue persists.

[a-and-m]

Hi,

thx for updating, I tried it with v191.0.11.0: Now I got no freezes anymore, but I cannot open the file at all. When double-clicking a log-file in the mentioned format nothing happens at all.

When disabling my custom formatter I can open the file, but I see no highlighting ("Log format not recognized"). When I disable all other formatters except my own custom formatter I cannot open the file.

[knah]

@a-and-m files not opening is likely a sign of an exception. If the exception reporter is not visible on UI (blinking red icon in lower-right corner), the exception can be found in the log. To find the log, use the "Help -> Show log in explorer" action. If you could attach the whole log, or find the relevant exception in it and post it here, it would likely help diagnose why files can't be opened.

[a-and-m]

Hi, thanks for the hint, the exception shows the following:

ERROR - tor.impl.FileEditorManagerImpl - Illegal pattern character 'T' 
java.lang.IllegalArgumentException: Illegal pattern character 'T'
    at java.text.SimpleDateFormat.compile(SimpleDateFormat.java:826)
    at java.text.SimpleDateFormat.initialize(SimpleDateFormat.java:634)
    at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:605)
    at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:580)
    at com.intellij.ideolog.util.IdeologDocumentContext.detectLogFileFormat(IdeologDocumentContext.kt:52)
    at com.intellij.ideolog.util.IdeologDocumentContext.getEventStartLine(IdeologDocumentContext.kt:118)
    at com.intellij.ideolog.util.IdeologDocumentContext.getEvent(IdeologDocumentContext.kt:74)

So probably the "T"-character in the time format must be escaped. How can you correctly escape that? (it literally means "T" and not any time value)

[knah]

In latest Ideolog version bad formats should be highlighted as such in the parsing pattern editor like in the screenshot below. If this doesn't happen, that's also a bug.

For the T character, according to Java docs, you should enclose it in single quotes, like this: yyyy-MM-dd'T'HH:mm:ss.SSSZ. Still, failing to open files is not a good behavior, thanks for providing the exception, I'll fix it once I get to it.

[a-and-m]

Yes, you were right, I also saw the "Illegal pattern character" message when editing the custom format now.

Thx for the info, I changed the time format now to this: yyyy-MM-dd'T'HH:mm:ss.SSS

But unfortunately, I now get the "log format not recognized" message when opening the file (but no exception).

For some reason the pattern is still not recognized....

[knah]

As I've said before, your format regexp seems to be fine. Does your log start with something that doesn't look like normal log messages, like one very long message (over 25 lines), or something that is not usual log messages?

The current format detection uses first 25 lines of the log and requires at least 5 of them to match with the message format. People have already complained about it and I'll probably redo that at some later point.

[a-and-m]

I had a log file that was shorter than 5 lines for testing reasons. I increased that now.

The content of my test log file looks like this currently:

[2019-05-13T00:00:00.005] [INFO] job.sendTfcCheckDropOutMails - ---> Starting job sendTfcCheckDropOutMails
[2019-05-13T00:00:10.008] [INFO] job.sendTfcCheckDropOutMails - Sending all mails for potential tfcCheckDropOuts that are considered as actual dropOuts
[2019-05-13T00:00:20.010] [INFO] job.processMailSequences - ---> Starting job processMailSequences
[2019-05-13T00:00:30.020] [INFO] job.processMailSequences - Sending all mails in sequences that must be sent by now
[2019-05-13T00:00:40.030] [INFO] job.processMailSequences - Sending all mails in sequences that must be sent by now
[2019-05-13T00:00:50.045] [INFO] job.processMailSequences - Sending all mails in sequences that must be sent by now
[2019-05-13T00:00:55.055] [INFO] job.processMailSequences - Sending all mails in sequences that must be sent by now
[2019-05-13T00:10:00.066] [INFO] job.processMailSequences - Sending all mails in sequences that must be sent by now
[2019-05-13T00:20:00.099] [INFO] job.processMailSequences - Sending all mails in sequences that must be sent by now

So I think it meets the needed requirements.

[rijnhard]

Same problem, I'm pretty good with regex and tested it specifically against java as well, then hacked in short compliant lines at the top of my log file and still it detects nothing.

# regex
Message pattern: ^\[([0-9\-T:.]{23})\] \[(\w+)\] ([\w.:]+) -(.*)$
Message start: ^\[
Time format: yyyy-MM-dd'T'HH:mm:ss.SSS

# log
[2019-05-18T20:43:24.193] [INFO] nope - sigh
[2019-05-18T20:43:24.193] [INFO] nope - sigh
[2019-05-18T20:43:24.193] [INFO] nope - sigh
[2019-05-18T20:43:24.193] [INFO] nope - sigh
[2019-05-18T20:43:24.193] [INFO] nope - sigh

[rijnhard]

Seems to be working now

[a-and-m]

Seems to be working now

Version 191.0.11.0 is not working for me: Still "Log format not recognized" when I activate only my own custom log format.

[a-and-m]

Still the same problem with version v191.0.12.0

[a-and-m]

Hi,

with version v192.0.12.0 the "Log format not recognized" error is gone, and the IDE is not freezing anymore.

But I am not sure if the log-file is correctly interpreted: I just see all log lines in green color, I do not see any indentation or the possibility to expand/collapse lines. Apart from the color there is no difference than opening the file in a regular text-editor in my case.

Is this behaviour expected? Aren't there any features missing with this? For example, is there the possibility to collapse log-entries that span over multiple lines into one line?

Thanks for your infos!

[knah]

Collapsing multi-line messages is a good feature idea, thanks! More advanced features (including filtering and coloring) are available from alt+enter menu. You can also customize highlighting in settings - green is the default color for messages with "info" severity.

[a-and-m]

Yes, collapsing multi-line log-entries would be very valuable for me, I very often have single log-entries that are huge and would love to collapse all of them into single lines.

Okay, then I guess this issue can be closed, thanks for your help!