Open lhrotk opened 4 months ago
Thank you for reporting this issue, I have investigated it and found the underlying bug. This is now fixed and will be released in the next update to open-datasets
package.
@lhrotk The patch for opendatasets==1.55.0.post1 is now released and this should work now
Hi, @lhrotk has the issue been resolved for you? Asking because I'm receiving the same error using 1.55.0.post1...
Hey @anliakho2, I am also getting this same error even after upgrading the azureml.opendatasets to 1.55.0.post1...
Hi, I encountered below error, I have azureml-opendatasets==1.55.0
Calling to_spark_dataframe() Traceback (most recent call last): File "/mnt/c/users/cruiseli/OneDrive - Microsoft/Desktop/workspace/SynapseML-Utils/test_aml.py", line 30, in
nyc_tlc_df2 = nyc_tlc.to_spark_dataframe()
File "/home/cruise/mambaforge/lib/python3.10/site-packages/azureml/opendatasets/accessories/_loggerfactory.py", line 139, in wrapper
return func(*args, **kwargs)
File "/home/cruise/mambaforge/lib/python3.10/site-packages/azureml/opendatasets/accessories/open_dataset_base.py", line 164, in to_spark_dataframe
return self._to_spark_dataframe()
File "/home/cruise/mambaforge/lib/python3.10/site-packages/azureml/opendatasets/accessories/open_dataset_base.py", line 305, in _to_spark_dataframe
return self._blob_accessor.get_spark_dataframe(
File "/home/cruise/mambaforge/lib/python3.10/site-packages/azureml/opendatasets/dataaccess/_blob_accessor.py", line 303, in get_spark_dataframe
paths = [wasab_format % (self._blob_container_name, self._blob_account_name,
File "/home/cruise/mambaforge/lib/python3.10/site-packages/azureml/opendatasets/dataaccess/_blob_accessor.py", line 304, in
self._get_relative_path(path)) for path in target_paths]
File "/home/cruise/mambaforge/lib/python3.10/site-packages/azureml/opendatasets/dataaccess/_blob_accessor.py", line 470, in _get_relative_path
if "blob.core.windows.net" in url:
TypeError: argument of type 'azureml.dataprep.rslex.StreamInfo' is not iterable
Code to reproduce: