coderabbitai[bot]
commented
1 week ago Walkthrough
The updates primarily enhance the handling of subject IDs across various modules, ensuring type compatibility and refining filtering logic. Key changes involve the FT_task_baseline.py, dataset_polars.py, and dataset_base.py files, improving the robustness of data processing. Additionally, a new script build_flat_reps.py for building flat representation datasets is introduced, along with multiple adjustments and new tests in the testing files.
Changes
| File(s) | Change Summary |
|---|---|
EventStream/baseline/FT_task_baseline.py |
Added overwrite_cache_filtered_task parameter, adjusted subject ID filtering logic using ESD.subject_id_dtype. |
EventStream/data/dataset_polars.py |
Updated subject ID dtype handling, revised n_events_per_subject calculation, adjusted inclusion targets filtering, added debug logs. |
EventStream/data/dataset_base.py |
Modified caching mechanism for reading and writing parameters, improved subject chunk processing for deep learning shard creation. |
EventStream/data/pytorch_dataset.py |
Updated _seeded_getitem method to ensure consistent data types for subject IDs. |
scripts/build_flat_reps.py |
New script introduced to build a flat representation dataset using a Hydra configuration file, including necessary imports and dataset processing steps. |
tests/test_e2e_runs.py |
Added imports for json and polars, introduced _test_dataset_output method, modified setUp and build_dataset methods for handling and asserting Parquet dataset contents. |
tests/data/test_pytorch_dataset.py |
Changed setUp method to convert a list of integers to strings in the shards dictionary construction. |
Sequence Diagram(s)
sequenceDiagram
participant User
participant Script as build_flat_reps.py
participant Dataset as EventStream.data.dataset_polars.Dataset
User ->> Script: Execute script with config file
Script ->> Dataset: Load dataset
Script ->> Dataset: Resolve cache parameters
Script ->> Dataset: Cache flat representation
Dataset ->> Script: Return success status
Script ->> User: Process completed[!TIP]
Early access features: enabled
We are currently testing the following features in early access: - **OpenAI `gpt-4o` model for code reviews and chat**: OpenAI claims that this model is better at understanding and generating code than the previous models. We seek your feedback over the next few weeks before making it generally available. Note: - You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file. - Please join our [Discord Community](https://discord.com/invite/GsXnASn26c) to provide feedback and report issues.
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
codecov[bot]
STILL IN PROGRESS. We just have failing test cases for now.