esloch
commented
1 year ago Hi @gabrielfmcoelho!
Thanks for reporting the issue. It has already been fixed in the latest PySUS release.
The "download" function will return a list of Parquet files and create them within the ~/pysus directory.
Additionally, I'd like to mention that the new PySUS release includes a module called "databases," allowing you to download CNES data directly from FTP. To see how it works, check out this example video: https://www.youtube.com/watch?v=vHNbnyxuNf4
If you have any more questions or encounter any issues, feel free to ask.
from pysus.online_data.CNES import download groups = ['LT', 'ST', 'DC' , 'SR', 'PF', 'EP'] state = "PI" years = [2022, 2023] months = [1,2,3,4,5,6,7,8,9,10,11,12]
for group in groups: df = download(group, state, years, months) print(df)
error: Traceback (most recent call last): File "/mnt/c/Users/gcmas/OneDrive/Área de Trabalho/S3 Biotech/Desenvolvimento/ETL Growth/CNES/Data/Temp/test.py", line 8, in
df = download(group, state, years, months)
File "/usr/local/lib/python3.10/dist-packages/pysus/online_data/CNES.py", line 50, in download
return FTP_Downloader('CNES').download(
File "/usr/local/lib/python3.10/dist-packages/pysus/online_data/init.py", line 430, in download
parquet_dir = self._dbfc_to_parquets(
File "/usr/local/lib/python3.10/dist-packages/pysus/online_data/init.py", line 585, in _dbfc_to_parquets
for d in self._stream_DBF(DBF(fpath, encoding='iso-8859-1', raw=True)):
File "/usr/local/lib/python3.10/dist-packages/dbfread-2.0.7-py3.10.egg/dbfread/dbf.py", line 122, in init
self._read_field_headers(infile)
File "/usr/local/lib/python3.10/dist-packages/dbfread-2.0.7-py3.10.egg/dbfread/dbf.py", line 224, in _read_field_headers
field = DBFField.unpack(sep + infile.read(DBFField.size - 1))
File "/usr/local/lib/python3.10/dist-packages/dbfread-2.0.7-py3.10.egg/dbfread/struct_parser.py", line 36, in unpack
items = zip(self.names, self.struct.unpack(data))
struct.error: unpack requires a buffer of 32 bytes
coelho@Coelho-G15:/mnt/c/Users/gcmas/OneDrive/Área de T