search

haskell / ghc-events

Library and tool for parsing .eventlog files from GHC
http://www.haskell.org/haskellwiki/ThreadScope
Other
33 stars 34 forks source link

Ix{Int}.index: Index (23637) out of range ((0,164)) #58

Open maoe opened 4 years ago

maoe commented 4 years ago

The eventlog file uploaded by @Unselfish at https://github.com/haskell/ThreadScope/issues/108#issuecomment-602936002 causes ghc-events inc to fail:

ghc-events: Ix{Int}.index: Index (23637) out of range ((0,164))

It was produced with GHC 8.6.5 on Windows.

mpickering commented 4 years ago

@maoe What does ghc-events inc show? Does it show a lot of 0: create thread 0 events?

maoe commented 4 years ago

Does it show a lot of 0: create thread 0 events?

No, not really.

unselfish.ghc-events-inc.txt

Unfortunately I don't have time to debug this today. Please feel free to take a look if you're interested.

maoe commented 4 years ago

@Unselfish I looked into https://github.com/haskell/ThreadScope/files/4376429/log4.txt briefly and found that the header of the file seemed broken.

If you look at hexdump of log4.txt, the beginning of the file looks like this:

  Offset: 0001 0203 0405 0607 0809 0A0B 0C0D 0E0F   
00000000: 6864 7262 6865 7462 6574 6200 0000 0004  hdrbhetbetb.....
00000010: 0000 000A 0D43 7265 6174 6520 7468 7265  .....Create.thre
00000020: 6164 0000 0000 6574 6500 6574 6200 0001  ad....ete.etb...

The first row (00000000) seems fine. The problem I think is the second row (00000010). The first 4 bytes of the row is a Word32 which represents the length of the next field, which is the string describing the event, "Create thread". The value is 0A here, which is 10 in decimal. This doesn't match the length of Create thread, which is 13! Actually if you truncate the 0A from the second row with a binary editor, ghc-events successfully parses the whole eventlog header. It still fails to parse the following events though.

While I haven't looked at the other eventlogs yet, I suspect your GHC or something (storage layer or memory modules?) is not working correctly. Can you try different versions of GHC or run the same program on a different machine, and see if the issue persists?

ttuegel commented 3 years ago

I see this error on Linux when the eventlog'd program is run with a large environment (about 69 kb, but I suspect the magic number is 2^16 bytes).

maoe commented 3 years ago

@ttuegel What do you mean by a large environment?

ttuegel commented 3 years ago

@maoe A large process environment, the environment variables. The ProgramEnv constructor of EventInfo.

maoe commented 3 years ago

@ttuegel I see. You're right that a payload of an eventlog message is limited to 2^16 bytes. See postPayloadSize in the RTS. This feature was removed as of GHC 8.8.1 though.

And I suppose this issue is something different because as I explained earlier the eventlog (https://github.com/haskell/ThreadScope/files/4376429/log4.txt) has a broken event header (rather than a payload).

mpickering commented 2 years ago

You can also run into this overflow if you have a command line longer than 2^16.