AttributeError: module 'pydantic.fields' has no attribute 'ModelField'

Hi,

My project directory : Items held:

PROJECT: /mnt/c/Users/pietr/stage_Walid/new_pykanto_test DATA: /mnt/c/Users/pietr/stage_Walid/new_pykanto_test/data RAW_DATA: /mnt/c/Users/pietr/stage_Walid/new_pykanto_test/data/raw/sociable_weaver SEGMENTED: /mnt/c/Users/pietr/stage_Walid/new_pykanto_test/data/segmented/sociable_weaver RESOURCES: /mnt/c/Users/pietr/stage_Walid/new_pykanto_test/resources REPORTS: /mnt/c/Users/pietr/stage_Walid/new_pykanto_test/reports FIGURES: /mnt/c/Users/pietr/stage_Walid/new_pykanto_test/reports/figures DATASET: /mnt/c/Users/pietr/stage_Walid/new_pykanto_test/data/datasets/sociable_weaver/sociable_weaver.db DATASET_ID: sociable_weaver SPECTROGRAMS: /mnt/c/Users/pietr/stage_Walid/new_pykanto_test/data/datasets/sociable_weaver/spectrograms

According to #27. I have wav files and corresponding csv annotations of units in RAW_DATA. In SEGMENTED I have json file and wav file for each vocalizations in csv file.

The problem that I have is when I instantiated KantoData I have the following error : AttributeError: module 'pydantic.fields' has no attribute 'ModelField'.

I use pykanto v0.1.5; pydantic v2.0.2; ray v2.5.1 on linux wsl 2 and jupyterlab 4.0.2.

There is the cell output :

Function '_get_wav_json_filedirs' took 0.0288 sec.
Chunkinfo(n_workers=20, len_iterable=179, n_chunks=36, chunksize=5, last_chunk=4)
Found 179 JSON files. They will be processed in 36 chunks of length 5.
2023-07-10 16:05:06,085 ERROR services.py:1207 -- Failed to start the dashboard , return code 1
2023-07-10 16:05:06,087 ERROR services.py:1232 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the last 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure' to find where the log file is.
2023-07-10 16:05:06,088 ERROR services.py:1276 -- 
The last 20 lines of /tmp/ray/session_2023-07-10_16-05-04_569204_166156/logs/dashboard.log (it contains the error message from the dashboard): 
  File "/home/walid/mambaforge/envs/new_pykanto/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/walid/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/dashboard/modules/log/log_manager.py", line 8, in <module>
    from ray.util.state.common import (
  File "/home/walid/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/util/state/__init__.py", line 1, in <module>
    from ray.util.state.api import (
  File "/home/walid/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/util/state/api.py", line 17, in <module>
    from ray.util.state.common import (
  File "/home/walid/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/util/state/common.py", line 120, in <module>
    @dataclass(init=True)
  File "/home/walid/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/pydantic/dataclasses.py", line 139, in dataclass
    assert init is False, 'pydantic.dataclasses.dataclass only supports init=False'
AssertionError: pydantic.dataclasses.dataclass only supports init=False
2023-07-10 16:05:06,252 INFO worker.py:1636 -- Started a local Ray instance.
(raylet) [2023-07-10 16:05:06,791 E 166387 166434] (raylet) agent_manager.cc:135: The raylet exited immediately because the Ray agent failed. The raylet fate shares with the agent. This can happen because the Ray agent was unexpectedly killed or failed. Agent can fail when
(raylet) - The version of `grpcio` doesn't follow Ray's requirement. Agent can segfault with the incorrect `grpcio` version. Check the grpcio version `pip freeze | grep grpcio`.
(raylet) - The agent failed to start because of unexpected error or port conflict. Read the log `cat /tmp/ray/session_latest/logs/dashboard_agent.log`. You can find the log file structure here https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure.
(raylet) - The agent is killed by the OS (e.g., out of memory).
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[10], line 1
----> 1 dataset = KantoData(DIRS, parameters=params)
      2 # dataset.data.head(3)

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/pykanto/dataset.py:123, in KantoData.__init__(self, DIRS, parameters, random_subset, overwrite_dataset, overwrite_data)
    120 self._get_wav_json_filedirs(random_subset=random_subset)
    121 # TODO: perform a field check for the json files and provide
    122 # feedback on any missing/wrong fields, or their value types.
--> 123 self._load_metadata()
    124 self._get_unique_ids()
    125 self._create_df()

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/pykanto/utils/compute.py:107, in timing.<locals>.wrap(*args, **kwargs)
    104 @wraps(f)
    105 def wrap(*args, **kwargs):
    106     start = time()
--> 107     output = f(*args, **kwargs)
    108     end = time()
    109     from pykanto.dataset import KantoData

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/pykanto/dataset.py:241, in KantoData._load_metadata(self)
    232     jsons = [
    233         _get_json(json)
    234         for json in with_pbar(
   (...)
    238         )
    239     ]
    240 else:
--> 241     jsons = _get_json_parallel(
    242         self._jsonfiles, verbose=self.parameters.verbose
    243     )
    244 self.metadata = {Path(json["wav_file"]).stem: json for json in jsons}
    246 # Match wav_file field with actual wav_file location for this dataset
    247 # Partially fixes nilomr/pykanto#12

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/pykanto/utils/io.py:171, in _get_json_parallel(lst, verbose)
    168 print_parallel_info(n, "JSON files", n_chunks, chunk_length)
    170 # Distribute with ray
--> 171 obj_ids = [_get_json_r.remote(i) for i in chunks]
    172 pbar = {"desc": "Loading JSON files", "total": n_chunks}
    173 jsons = [obj_id for obj_id in with_pbar(to_iterator(obj_ids), **pbar)]

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/pykanto/utils/io.py:171, in <listcomp>(.0)
    168 print_parallel_info(n, "JSON files", n_chunks, chunk_length)
    170 # Distribute with ray
--> 171 obj_ids = [_get_json_r.remote(i) for i in chunks]
    172 pbar = {"desc": "Loading JSON files", "total": n_chunks}
    173 jsons = [obj_id for obj_id in with_pbar(to_iterator(obj_ids), **pbar)]

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/remote_function.py:133, in RemoteFunction.__init__.<locals>._remote_proxy(*args, **kwargs)
    131 @wraps(function)
    132 def _remote_proxy(*args, **kwargs):
--> 133     return self._remote(args=args, kwargs=kwargs, **self._default_options)

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/util/tracing/tracing_helper.py:306, in _tracing_task_invocation.<locals>._invocation_remote_span(self, args, kwargs, *_args, **_kwargs)
    304     if kwargs is not None:
    305         assert "_ray_trace_ctx" not in kwargs
--> 306     return method(self, args, kwargs, *_args, **_kwargs)
    308 assert "_ray_trace_ctx" not in kwargs
    309 tracer = _opentelemetry.trace.get_tracer(__name__)

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/remote_function.py:422, in RemoteFunction._remote(self, args, kwargs, **task_options)
    419 if self._decorator is not None:
    420     invocation = self._decorator(invocation)
--> 422 return invocation(args, kwargs)

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/remote_function.py:391, in RemoteFunction._remote.<locals>.invocation(args, kwargs)
    387 if worker.mode == ray._private.worker.LOCAL_MODE:
    388     assert (
    389         not self._is_cross_language
    390     ), "Cross language remote function cannot be executed locally."
--> 391 object_refs = worker.core_worker.submit_task(
    392     self._language,
    393     self._function_descriptor,
    394     list_args,
    395     name if name is not None else "",
    396     num_returns,
    397     resources,
    398     max_retries,
    399     retry_exceptions,
    400     retry_exception_allowlist,
    401     scheduling_strategy,
    402     worker.debugger_breakpoint,
    403     serialized_runtime_env_info or "{}",
    404 )
    405 # Reset worker's debug context from the last "remote" command
    406 # (which applies only to this .remote call).
    407 worker.debugger_breakpoint = b""

File python/ray/_raylet.pyx:2898, in ray._raylet.CoreWorker.submit_task()

File python/ray/_raylet.pyx:2902, in ray._raylet.CoreWorker.submit_task()

File python/ray/_raylet.pyx:671, in ray._raylet.prepare_args_and_increment_put_refs()

File python/ray/_raylet.pyx:662, in ray._raylet.prepare_args_and_increment_put_refs()

File python/ray/_raylet.pyx:708, in ray._raylet.prepare_args_internal()

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/_private/worker.py:618, in Worker.get_serialization_context(self)
    616         context_map[job_id] = context_map.pop(JobID.nil())
    617     else:
--> 618         context_map[job_id] = serialization.SerializationContext(self)
    619 return context_map[job_id]

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/_private/serialization.py:151, in SerializationContext.__init__(self, worker)
    145     return ObjectRefGenerator, (obj._refs,)
    147 self._register_cloudpickle_reducer(
    148     ObjectRefGenerator, object_ref_generator_reducer
    149 )
--> 151 serialization_addons.apply(self)

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/util/serialization_addons.py:58, in apply(serialization_context)
     56 @DeveloperAPI
     57 def apply(serialization_context):
---> 58     register_pydantic_serializer(serialization_context)
     59     register_starlette_serializer(serialization_context)
     61     if sys.platform != "win32":

File ~/mambaforge/envs/new_pykanto/lib/python3.9/site-packages/ray/util/serialization_addons.py:21, in register_pydantic_serializer(serialization_context)
     16     return
     18 # Pydantic's Cython validators are not serializable.
     19 # https://github.com/cloudpipe/cloudpickle/issues/408
     20 serialization_context._register_cloudpickle_serializer(
---> 21     pydantic.fields.ModelField,
     22     custom_serializer=lambda o: {
     23         "name": o.name,
     24         # outer_type_ is the original type for ModelFields,
     25         # while type_ can be updated later with the nested type
     26         # like int for List[int].
     27         "type_": o.outer_type_,
     28         "class_validators": o.class_validators,
     29         "model_config": o.model_config,
     30         "default": o.default,
     31         "default_factory": o.default_factory,
     32         "required": o.required,
     33         "alias": o.alias,
     34         "field_info": o.field_info,
     35     },
     36     custom_deserializer=lambda kwargs: pydantic.fields.ModelField(**kwargs),
     37 )

AttributeError: module 'pydantic.fields' has no attribute 'ModelField'

nilomr / pykanto

AttributeError: module 'pydantic.fields' has no attribute 'ModelField' #28