pytorch / data

A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.
BSD 3-Clause "New" or "Revised" License
1.12k stars 149 forks source link

update worker keys and add test #1250

Closed andrewkho closed 4 months ago

andrewkho commented 4 months ago

Users [internal] are occasionally using json.dumps to serialize the state_dict when they know the state_dicts are JSON serializable. Ignoring the fact this won't work in general, this fails because json spec does not allow int properties (string required) which we're using for _worker_states in the state dict. This PR changes the format of the state_dict to use strings instead, ie _worker_states: {0: ..., 1: ...} -> _worker_states: {"worker_0": ..., "worker_1": ...}

Changes

facebook-github-bot commented 4 months ago

@andrewkho has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 4 months ago

@andrewkho merged this pull request in pytorch/data@c1f8b66f9b6e87625541f3f915c7822245d609b8.