neptune-ai / neptune-client

📘 The experiment tracker for foundation model training
https://neptune.ai
Apache License 2.0
580 stars 63 forks source link

feat: Added `expand` flag to `stringify_unsupported()` to expand series #1862

Closed SiddhantSadangi closed 4 days ago

SiddhantSadangi commented 2 months ago

Before submitting checklist

stringify_unsupported() does not currently expand lists, casting them to string instead.

For dicts that contain lists as values, as shown below

nested_dict = {
    "name": "John Doe",
    "age": 30,
    "is_employee": True,
    "address": {"street": "123 Main St", "city": "Anytown", "postal_code": "12345"},
    "projects": [
        {"name": "Project Alpha", "status": "completed", "budget": 100000},
        {"name": "Project Beta", "status": "in progress", "budget": 150000},
    ],
    "skills": ["Python", "Data Analysis", "Machine Learning"],
    "employment_history": [
        {"company": "Company A", "position": "Data Scientist", "years": 3},
        {"company": "Company B", "position": "Software Engineer", "years": 2},
    ],
}

this leads to keys like projects, skills, and employment_history being logged as a string image Example

This makes it harder to find relevant information if it is nested as a list in a dict, and also makes it a lot harder to download such metadata from Neptune, as it would be downloaded as a string.

This PR adds an optional expand flag to stringify_unsupported() to expand series like lists, sets, and tuples thereby storing nested values in their proper folder structure within enumerated folders, making it easier to both search for metadata and download it. image Example

codecov[bot] commented 1 week ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 75.59%. Comparing base (7da7535) to head (c0f65f2). Report is 1 commits behind head on dev/1.x.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## dev/1.x #1862 +/- ## =========================================== - Coverage 77.56% 75.59% -1.97% =========================================== Files 303 303 Lines 15382 15366 -16 =========================================== - Hits 11931 11616 -315 - Misses 3451 3750 +299 ``` | [Flag](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | Coverage Δ | | |---|---|---| | [e2e](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `?` | | | [e2e-management](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `?` | | | [e2e-s3](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `?` | | | [e2e-s3-gcs](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `?` | | | [macos](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `75.27% <100.00%> (-2.02%)` | :arrow_down: | | [py3.10](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `?` | | | [py3.11](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `?` | | | [py3.12](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `?` | | | [py3.8](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `75.59% <100.00%> (-1.93%)` | :arrow_down: | | [py3.9](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `?` | | | [ubuntu](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `75.41% <100.00%> (-2.02%)` | :arrow_down: | | [unit](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `75.59% <100.00%> (-0.04%)` | :arrow_down: | | [windows](https://app.codecov.io/gh/neptune-ai/neptune-client/pull/1862/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai) | `74.50% <100.00%> (-1.97%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=neptune-ai#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.