Fix preprocessing code for LLM

[tanzir5]

This document explains the pipeline of the LLM. https://docs.google.com/document/d/1bWVtvZi-harAbY8wBhcyLyLEEJGMsXbIS5vDZbrusaA/edit

• [ ] Add the "calculate age" functionality here. We need access to the background file as well as RINPERSOON column in the given file to calculate this, both of which are reasonable expectations. https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/8182bd539d9285dccaa456fde9f7bbe3f13602e1/pop2vec/llm/src/new_code/preprocess_data.py#L104
• [ ] Change the quantization process to percentile calculation. https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/8182bd539d9285dccaa456fde9f7bbe3f13602e1/pop2vec/llm/src/new_code/preprocess_data.py#L48
• [ ] Tidy up the code [E.g., better handling of time_key, docstring for functions, etc] https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/main/pop2vec/llm/src/new_code/preprocess_data.py
• [ ] Instead of json, store sequences in either a hdf5 file or a sqlite database. This will make it easier to query the sequence given the RINPERSOON and make file-handling easier in general. https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/main/pop2vec/llm/src/new_code/create_life_seq_jsons.py
• [ ] Tidy up the code. https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/main/pop2vec/llm/src/new_code/create_life_seq_jsons.py https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/main/pop2vec/llm/src/new_code/create_person_dict.py
• [ ] Handle vocab creation and loading. Right now we do not have vocab loading functionality. Rather we always create the vocab from the scratch. https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/main/pop2vec/llm/src/new_code/create_person_dict.py https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/main/pop2vec/llm/src/new_code/create_life_seq_jsons.py
• [ ] Write shuffling logic based on the previous task. https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/8182bd539d9285dccaa456fde9f7bbe3f13602e1/pop2vec/llm/src/new_code/pipeline.py#L292
• [ ] Fix data loading based on how event sequences are stored. https://github.com/odissei-lifecourse/life-sequencing-dutch/blob/8182bd539d9285dccaa456fde9f7bbe3f13602e1/pop2vec/llm/src/new_code/pipeline.py#L272
• [ ] Add functionality for adding events to an existing event sequence.