Support different suffixes for parquet files

[gschwend]

Please add .parquet and .parq to the list of valid file extensions for parquet files. These two options were used in DP0.1 and DP0.2 catalog files, respectively.

[gschwend]

@eacharles I can contribute with this piece of code. Just add me to this repo, if you agree, so I can assign the issue to myself.

[eacharles]

Hi Julia,

You were already on the team, but only had write access, I upgraded that to “maintain” access to let you manage issues.

-e

On Mar 31, 2023, at 11:34 AM, Julia Gschwend @.***> wrote:

@eacharles https://github.com/eacharles I can contribute with this piece of code. Just add me to this repo, if you agree, so I can assign the issue to myself.

— Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/tables_io/issues/61#issuecomment-1492426055, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRIGIT357HU55HIDRU6OXDW64PTHANCNFSM6AAAAAAWO7XMFY. You are receiving this because you were mentioned.

[gschwend]

@eacharles can you double-check if the upgrade you did really worked? I am trying to push a new branch with the commits but it returns as permission denied:

(tables_io) julia@dell-inspiron ~/github/tables_io/docs (issue/61/parquet-suffixes) $ git push --set-upstream origin issue/61/parquet-suffixes
Username for 'https://github.com': gschwend
Password for 'https://gschwend@github.com': 
remote: Permission to LSSTDESC/tables_io.git denied to gschwend.
fatal: unable to access 'https://github.com/LSSTDESC/tables_io.git/': The requested URL returned error: 403

[eacharles]

You are right, I added upgraded you on RAIL. Ok, just did tables_io too.

-e

On Mar 31, 2023, at 2:24 PM, Julia Gschwend @.***> wrote:

@eacharles https://github.com/eacharles can you double-check if the upgrade you did really worked? I am trying to push a new branch with the commits but it returns as permission denied:

(tables_io) @. ~/github/tables_io/docs (issue/61/parquet-suffixes) $ git push --set-upstream origin issue/61/parquet-suffixes Username for 'https://github.com': gschwend Password for @.': remote: Permission to LSSTDESC/tables_io.git denied to gschwend. fatal: unable to access 'https://github.com/LSSTDESC/tables_io.git/': The requested URL returned error: 403 — Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/tables_io/issues/61#issuecomment-1492628429, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRIGIUKJKNAID2XXO4MH5TW65DSVANCNFSM6AAAAAAWO7XMFY. You are receiving this because you were mentioned.

[gschwend]

Thanks!

[eacharles]

closed with #62

[hdante]

Hello, is it possible to reopen this issue ? I'm not being able to open files with the extensions ".parq" and ".parquet" because read(), a FileNotFoundError exception is thrown (but the file described in the exception message ends up with the ".pq" extension again):

>>> f = tables_io.read('objectTable_tract_3635_DC2_2_2i_runs_DP0_2_v23_0_1_PREOPS-905_step3_12_20220218T172932Z.parq')
Traceback (most recent call last):
  File "/home/hdante/Trabalho/LineA/local/lib/python3.10/site-packages/tables_io/ioUtils.py", line 792, in readPqToDataFrames
    dataframes[key] = readPqToDataFrame(f"{basepath}{key}.pq")
  File "/home/hdante/Trabalho/LineA/local/lib/python3.10/site-packages/tables_io/ioUtils.py", line 747, in readPqToDataFrame
    return pd.read_parquet(filepath, engine='pyarrow')
  File "/home/hdante/Trabalho/LineA/local/lib/python3.10/site-packages/pandas/io/parquet.py", line 503, in read_parquet
    return impl.read(
  File "/home/hdante/Trabalho/LineA/local/lib/python3.10/site-packages/pandas/io/parquet.py", line 244, in read
    path_or_handle, handles, kwargs["filesystem"] = _get_path_or_handle(
  File "/home/hdante/Trabalho/LineA/local/lib/python3.10/site-packages/pandas/io/parquet.py", line 102, in _get_path_or_handle
    handles = get_handle(
  File "/home/hdante/Trabalho/LineA/local/lib/python3.10/site-packages/pandas/io/common.py", line 865, in get_handle
    handle = open(handle, ioargs.mode)
FileNotFoundError: [Errno 2] No such file or directory: 'objectTable_tract_3635_DC2_2_2i_runs_DP0_2_v23_0_1_PREOPS-905_step3_12_20220218T172932Z.pq'

[gschwend]

Yes, I am going to take a look.