Resolve catalog issues

[jzwart]

Builds have been failing for a while, but I haven't touched it because there seemed to be a lot of activity going on in https://github.com/eco4cast/neon4cast-ci and https://github.com/eco4cast/stac4cast and I wanted to let that settle. Now I'm getting into trying to solve the catalog issues after work that @addelany has put into stac4cast.

Currently, the catalog build is failing for usgsrc4cast when building the group variables - see here. Specifically, it fails when trying to apply the build_table_columns_full_bucket function to the forecast dataframe when using the ToString() function here . I don't think I've seen this ToString() function or syntax before, @cboettig @addelany could you provide some more context for how this should work?

[addelany]

Hi @jzwart , sorry for the constant changes with the stac4cast and other neon4cast functions. We are still doing development and testing, so these functions (especially stac4cast) are not yet concrete.

Looking at your specific issue, and it seems like there may be a mix up of the forecast_theme_df and the forecast_data_df objects for the group variable build. The first argument of the build_group_variables() function should be forecast_theme_df, which accesses just the schema of the data. Here is an example where I think you need to make that change.

The forecast_theme_df object is created by using arrow::open_dataset() without a closing collect() statement (see here). ToString() is a function within arrow that converts the schema output from a list to a string, which was easier to use for this case.

There may be other issues that you run into due to recent changes with the catalog build code, so please reach out with more questions and issues.

[jzwart]

Thanks @addelany !

ugh, it's always something simple like that - thanks for providing another set of eyes and for all your work on stac4cast and neon4cast-ci! This got me over this hump and I'll plug away on the rest of the catalog ci fixes, and ping you if needed.