I saw a new warning this morning, when trying out a DP0 import:
/astro/users/mmd11/git/hipscat-import/src/hipscat_import/catalog/map_reduce.py:289: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
dataframe["Norder"] = np.full(rows_written, fill_value=healpix_pixel.order, dtype=np.uint8)
/astro/users/mmd11/git/hipscat-import/src/hipscat_import/catalog/map_reduce.py:290: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
dataframe["Dir"] = np.full(rows_written, fill_value=healpix_pixel.dir, dtype=np.uint64)
/astro/users/mmd11/git/hipscat-import/src/hipscat_import/catalog/map_reduce.py:291: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
dataframe["Npix"] = np.full(rows_written, fill_value=healpix_pixel.pixel, dtype=np.uint64)
We should try out the suggested pd.concat(axis=1) approach.
Before submitting
Please check the following:
[ ] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
[x] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
[x] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
Bug report
I saw a new warning this morning, when trying out a DP0 import:
We should try out the suggested
pd.concat(axis=1)
approach.Before submitting Please check the following: