With the current layout of Token the size is 32 bytes. The TokenType enum should be put last for better alignment. I've pushed a change to #1 that also makes the TokenType explicitly the size of a byte (uint8_t).
Moving the TokenType to the bottom of the struct reduces the size to 24 bytes. But, we can actually stay at 24 bytes and increase the mLine allocation to a uint32_t. This lets us have peace of mind and not have to inform the user that their files are limited to only 65,536 lines.
So, we can now ingest larger files and we don't even increase the token size on most platforms.
arturocepeda
commented
2 years ago
With the current layout of Token the size is 32 bytes. The TokenType enum should be put last for better alignment. I've pushed a change to #1 that also makes the TokenType explicitly the size of a byte (uint8_t).
Moving the TokenType to the bottom of the struct reduces the size to 24 bytes. But, we can actually stay at 24 bytes and increase the mLine allocation to a uint32_t. This lets us have peace of mind and not have to inform the user that their files are limited to only 65,536 lines.
So, we can now ingest larger files and we don't even increase the token size on most platforms.
This can be closed if #1 is pulled.