Closed ghost closed 5 years ago
jonescc
commented
5 years ago GeoServer (2.8) has an option for ignoring invalid layers when generating GetCapabilites documents (seem to recall @pmbohm mentioning this in the past):
We should probably add a check for invalid layers when merging rc config into prod to ensure we pick up any problem layers before they are deployed to prod - we may also be able to add a check to pull requests to validate their correctness before merging.
jonescc
commented
5 years ago Make that we can definitely add validation checks on pull requests using travis which allows arbitrary scripts to be run. One possible script could be the one @pmbohm identified in https://github.com/aodn/geoserver-config/issues/435 although we would obviously need to cleanup the current issues first.
pmbohm
commented
5 years ago Yes hopefully its some configuration flag we can set for all Geoservers
ghost
commented
5 years ago I enabled the option in PO box, and it generates the following diff: https://github.com/aodn/geoserver-config/pull/577
Yes updating the 'geoserver_config_prod_merge_test' job in Jenkins to do more specific validation of the layer.xml files is a good idea.
I haven't been able to find a schema describing the layer.xml files, despite the documentation stating that it is required, so either I'm missing something of the validation is an internal Java thing when it loads the config.
ghost
commented
5 years ago I've looked at the Perl utility and doesn't seem to work any more with current modules/Perl version:
10:37:34 (master) ~/git/landryb/geoserver-datadir-checker$ PERL_USE_UNSAFE_INC=1 perl check.pl ../../aodn/geoserver-config
Can't locate object method "host" via package "URI::_generic" at /usr/share/perl5/URI/WithBase.pm line 53.I started looking at a Python solution but it was smaller in scope, only checking styles. Depends how broad we wan't the checking, but the Perl project might end up creating more work than it saves!
jonescc
commented
5 years ago No I can't find a schema for layer.xml either. Looks like its determined by the java code.
Re the perl script, yes I should have looked at the last commit date - no changes in 3 years so likely not maintained and we'd be better off writing something specific for our requirements.
ghost
commented
5 years ago Knocked up a script to validate layer XML styles, but could be extended if specific validation rules are identified: https://github.com/aodn/utilities/pull/60
Gives output like:
$ ./validate_geoserver_config.py ~/geoserver-config
VALID LAYERS COUNT: 241
INVALID LAYERS COUNT: 66
INVALID LAYERS:
srs_oc_so_johnson_chl_mo_seawifs_url: status='INVALID' reason='missing defaultStyle tag'
csiro_cars_monthly_url: status='INVALID' reason='references nonexistent style: StyleInfoImpl--396ecd48:155a40f3d82:-7ffc'
csiro_oa_reconstruction_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_3d_day_url: status='INVALID' reason='missing defaultStyle tag'
aodn_mhl_wave_dm_timeseries_data: status='INVALID' reason='references nonexistent style: StyleInfoImpl-5aa01124:15233403c30:-7ffc'
srs_ghrsst_l3s_6d_dn_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3c_1d_S_day_n19_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_sst_1d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
acorn_wind_qc_cof_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_nanop_brewin2010at_pft_1d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
aodn_bom_sea_level_timeseries_data: status='INVALID' reason='references nonexistent style: StyleInfoImpl-5aa01124:15233403c30:-7ffc'
srs_oc_so_johnson_chl_8d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_6d_day_url: status='INVALID' reason='missing defaultStyle tag'
aodn_nt_sattag_hawksbill_summary_data: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3c_1d_day_n19_url: status='INVALID' reason='missing defaultStyle tag'
bathy_ppb_deakin_lower_res_url: status='INVALID' reason='missing defaultStyle tag'
acorn_hourly_avg_qc_timeseries_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3u_n19_url: status='INVALID' reason='missing defaultStyle tag'
aodn_des_qld_wave_dm_timeseries_data: status='INVALID' reason='references nonexistent style: StyleInfoImpl-5aa01124:15233403c30:-7ffc'
srs_ghrsst_l3s_1m_dn_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_1dS_dn_url: status='INVALID' reason='missing defaultStyle tag'
bathy_ppb_deakin_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_1m_day_url: status='INVALID' reason='missing defaultStyle tag'
acorn_wind_qc_rot_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_14d_dn_url: status='INVALID' reason='missing defaultStyle tag'
srs_sst_l3s_1m_ngt_gridded_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_npp_vgpm_epp_gsm_1d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_14d_ngt_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_so_johnson_chl_mo_aqua_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_1d_ngt_url: status='INVALID' reason='missing defaultStyle tag'
auv_trajectory_b_data: status='INVALID' reason='missing defaultStyle tag'
acorn_wave_site_qc_rot_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_14d_day_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3c_3d_ngt_n19_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_1d_dn_url: status='INVALID' reason='missing defaultStyle tag'
acorn_wind_qc_sag_url: status='INVALID' reason='missing defaultStyle tag'
srs_sst_l3s_1m_dn_gridded_url: status='INVALID' reason='missing defaultStyle tag'
aodn_bom_wave_dm_timeseries_data: status='INVALID' reason='references nonexistent style: StyleInfoImpl-5aa01124:15233403c30:-7ffc'
srs_ghrsst_l3s_3d_ngt_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_1d_day_url: status='INVALID' reason='references nonexistent style: StyleInfoImpl--28ae9436:15722231f36:-7ffb'
acorn_wave_site_qc_sag_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_3d_dn_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3c_3d_day_n19_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_chl_gsm_1d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
acorn_wave_site_qc_cof_url: status='INVALID' reason='missing defaultStyle tag'
acorn_wind_qc_cbg_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3u_S_n19_url: status='INVALID' reason='missing defaultStyle tag'
csiro_cars_weekly_url: status='INVALID' reason='references nonexistent style: StyleInfoImpl--396ecd48:155a40f3d82:-7ffc'
sstaars_daily_fit_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_nanop_brewin2012in_pft_1d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_picop_brewin2012in_pft_1d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_npp_vgpm_epp_oc3_1d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_1m_ngt_url: status='INVALID' reason='missing defaultStyle tag'
acorn_hourly_avg_nonqc_timeseries_url: status='INVALID' reason='missing defaultStyle tag'
srs_sst_l3s_1m_day_gridded_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_picop_brewin2010at_pft_1d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3c_1d_S_ngt_n19_url: status='INVALID' reason='missing defaultStyle tag'
aodn_dot_wa_wave_dm_timeseries_data: status='INVALID' reason='references nonexistent style: StyleInfoImpl-5aa01124:15233403c30:-7ffc'
srs_ghrsst_l3s_6d_ngt_url: status='INVALID' reason='missing defaultStyle tag'
aodn_nt_sattag_oliveridley_summary_data: status='INVALID' reason='missing defaultStyle tag'
srs_oc_so_johnson_chl_8d_seawifs_url: status='INVALID' reason='missing defaultStyle tag'
srs_oc_chl_oc3_1d_aqua_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3c_1d_ngt_n19_url: status='INVALID' reason='missing defaultStyle tag'
acorn_wave_site_qc_cbg_url: status='INVALID' reason='missing defaultStyle tag'
srs_sst_l3s_1m_southern_dn_gridded_url: status='INVALID' reason='missing defaultStyle tag'
srs_ghrsst_l3s_1mS_dn_url: status='INVALID' reason='missing defaultStyle tag'
Process finished with exit code 0If it returns an error when issues are found, then we can include it in the repo and get travis to run it for pull requests to be validated.
ghost
commented
5 years ago Oops, that was the whole point after all. It now exits with and error if any invalid layers found.
Not sure how anything works with 66 invalid ones currently!
ggalibert
commented
5 years ago Does this problem still exist @lwgordonimos ?
ghost
commented
5 years ago The immediate problem was resolved by upgrading Geoserver, but it is still desirable to do more validation before deployment (perhaps even before merging), in order to pick up issues earlier.
There is a recurring issue which breaks GetCapabilities responses from Geoserver relating to layers missing a defaultStyle in their layer.xml.
For example:
https://github.com/search?q=org%3Aaodn+Unable+to+transform+referenced+envelope%2C+crs+has+not+yet+been+provided.&type=Issues
This is a list of all layer.xml files without this tag:
Because this now breaks Geowebcache if the GetCapabilities document is broken, we should take steps to make sure this doesn't happen.
Discuss!