search

ECCO-GROUP / ECCO-v4-Configurations

This repository contains documentation (doc/) and model configuration files (code/, namelist/) for official releases of the ECCO version 4 ocean and sea-ice state estimates. Model configuration files allow users to reproduce the state estimate or conduct new simulation experiments.
16 stars 19 forks source link

Compilation of ECCOv4r4 flux-forced adjoint fails with "failed to convert" error #148

Closed emmomp closed 2 years ago

emmomp commented 2 years ago

I'm attempting to compile the r4 flux-forced version on the ARCHER2 machine in the UK. I can successfully compile the 'regular' r4 version. I have an up-to-date version of the code and using the following command:

export MITGCM_ROOTDIR=/home/n01/n01/emmomp/MITgcm
export PATH=$MITGCM_ROOTDIR/tools:$PATH
export MITGCM_OPT=$MITGCM_ROOTDIR/tools/build_options/dev_linux_amd64_cray_archer2

module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel

genmake2 -ieee -mpi -mods=../code_ff_ad -of=$MITGCM_OPT
make depend
make adtaf
make adall`

Here's my Makefile for reference. The flux forced build is failing at the TAF stage with the following error:

ftn -o mitgcmuv_ad -em -ef -dynamic -h pic -O0 -hfp0 ad_taf_output.o addummy_in_stepping.o diagnostics_fill_state.o diagnostics_main_init.o do_the_model_io.o ecco_check.o ecco_cost_init_barfiles.o ecco_readparms.o exf_check.o exf_init_fixed.o exf_monitor.o exf_readparms.o exf_summary.o mdsio_write_meta.o pabar_output.o profiles_init_fixed.o stergloh_output.o active_file_ad.o active_file_control.o active_file_control_slice.o active_file.o active_file_gen_ad.o active_file_gen.o active_file_gen_g.o active_file_g.o active_file_loc_ad.o active_file_loc.o active_file_loc_g.o adautodiff_whtapeio_sync.o addamp_adj.o addummy_in_dynamics.o adopen_adclose.o adread_adwrite.o adzero_adj.o autodiff_check.o autodiff_findunit.o autodiff_inadmode_set_ad.o autodiff_inadmode_set.o autodiff_inadmode_set_g.o autodiff_inadmode_unset_ad.o autodiff_inadmode_unset.o autodiff_inadmode_unset_g.o autodiff_ini_model_io.o autodiff_readparms.o autodiff_whtapeio_sync.o copy_ad_uv_outp.o copy_advar_outp.o damp_adj.o dummy_in_dynamics.o dummy_in_stepping.o g_dummy_in_dynamics.o g_dummy_in_stepping.o global_max_ad.o global_sum_ad.o global_sum_tile_ad.o myactivefunction_ad.o myactivefunction.o zero_adj.o cal_daysformonth.o cal_dayspermonth.o cal_init_fixed.o cal_monthsforyear.o cal_monthsperyear.o cal_readparms.o cal_set.o cal_stepsforday.o cal_stepsperday.o cal_summary.o cal_weekday.o cost_check.o cost_dependent_init.o cost_depth.o cost_final_restore.o cost_final_store.o cost_readparms.o adctrl_bound.o ctrl_bound.o ctrl_check.o ctrl_hfacc_ini.o ctrl_init_ctrlvar.o ctrl_init.o ctrl_init_wet.o ctrl_mask_set_xz.o ctrl_mask_set_yz.o ctrl_pack.o ctrl_readparms.o ctrl_set_fname.o ctrl_set_globfld_xy.o ctrl_set_globfld_xyz.o ctrl_set_globfld_xz.o ctrl_set_globfld_yz.o ctrl_set_pack_xy.o ctrl_set_pack_xyz.o ctrl_set_pack_xz.o ctrl_set_pack_yz.o ctrl_set_unpack_xy.o ctrl_set_unpack_xyz.o ctrl_set_unpack_xz.o ctrl_set_unpack_yz.o ctrl_summary.o ctrl_unpack.o optim_readparms.o chksum_tiled.o debug_call.o debug_cs_corner_uv.o debug_enter.o debug_fld_stats_rl.o debug_fld_stats_rs.o debug_leave.o debug_msg.o debug_stats_rl.o debug_stats_rs.o fill_in_corners_rl.o write_fullarray_rl.o write_fullarray_rs.o diag_calc_psivel.o diag_cg2d.o diagnostics_addtolist.o diagnostics_calc_phivel.o diagnostics_check.o diagnostics_clear.o diagnostics_fill.o diagnostics_fill_field.o diagnostics_fill_rs.o diagnostics_fract_fill.o diagnostics_ini_io.o diagnostics_init_early.o diagnostics_init_fixed.o diagnostics_init_varia.o diagnostics_interp_p2p.o diagnostics_interp_vert.o diagnostics_list_check.o diagnostics_mnc_out.o diagnostics_out.o diagnostics_readparms.o diagnostics_read_pickup.o diagnostics_scale_fill.o diagnostics_scale_fill_rs.o diagnostics_set_calc.o diagnostics_setdiag.o diagnostics_set_levels.o diagnostics_set_pointers.o diagnostics_status_error.o diagnostics_sum_levels.o diagnostics_summary.o diagnostics_switch_onoff.o diagnostics_utils.o diagnostics_write.o diagnostics_write_pickup.o diagstats_ascii_out.o diagstats_calc.o diagstats_clear.o diagstats_close_io.o diagstats_fill.o diagstats_global.o diagstats_ini_io.o diagstats_local.o diagstats_mnc_out.o diagstats_others_calc.o diagstats_output.o diagstats_setdiag.o diagstats_set_pointers.o diagstats_set_regions.o diag_vegtile_fill.o ecco_cost_init_fixed.o ecco_cost_summary.o ecco_cost_weights.o ecco_summary.o exch2_3d_r4.o exch2_3d_r8.o exch2_ad_get_r41.o exch2_ad_get_r42.o exch2_ad_get_r81.o exch2_ad_get_r82.o exch2_ad_get_rl1.o exch2_ad_get_rl2.o exch2_ad_get_rs1.o exch2_ad_get_rs2.o exch2_ad_put_r41.o exch2_ad_put_r42.o exch2_ad_put_r81.o exch2_ad_put_r82.o exch2_ad_put_rl1.o exch2_ad_put_rl2.o exch2_ad_put_rs1.o exch2_ad_put_rs2.o exch2_check_depths.o exch2_get_r41.o exch2_get_r42.o exch2_get_r81.o exch2_get_r82.o exch2_get_rl1.o exch2_get_rl2.o exch2_get_rs1.o exch2_get_rs2.o exch2_get_scal_bounds.o exch2_get_uv_bounds.o exch2_put_r41.o exch2_put_r42.o exch2_put_r81.o exch2_put_r82.o exch2_put_rl1.o exch2_put_rl2.o exch2_put_rs1.o exch2_put_rs2.o exch2_r41_cube_ad.o exch2_r41_cube.o exch2_r42_cube_ad.o exch2_r42_cube.o exch2_r81_cube_ad.o exch2_r81_cube.o exch2_r82_cube_ad.o exch2_r82_cube.o exch2_recv_r41.o exch2_recv_r42.o exch2_recv_r81.o exch2_recv_r82.o exch2_recv_rl1.o exch2_recv_rl2.o exch2_recv_rs1.o exch2_recv_rs2.o exch2_rl1_cube_ad.o exch2_rl1_cube.o exch2_rl2_cube_ad.o exch2_rl2_cube.o exch2_rs1_cube_ad.o exch2_rs1_cube.o exch2_rs2_cube_ad.o exch2_rs2_cube.o exch2_s3d_r4.o exch2_s3d_r8.o exch2_s3d_rl.o exch2_s3d_rs.o exch2_send_r41.o exch2_send_r42.o exch2_send_r81.o exch2_send_r82.o exch2_send_rl1.o exch2_send_rl2.o exch2_send_rs1.o exch2_send_rs2.o exch2_sm_3d_r4.o exch2_sm_3d_r8.o exch2_sm_3d_rs.o exch2_uv_3d_r4.o exch2_uv_3d_r8.o exch2_uv_agrid_3d_r4.o exch2_uv_agrid_3d_r8.o exch2_uv_bgrid_3d_r4.o exch2_uv_bgrid_3d_r8.o exch2_uv_bgrid_3d_rl.o exch2_uv_bgrid_3d_rs.o exch2_uv_cgrid_3d_r4.o exch2_uv_cgrid_3d_r8.o exch2_uv_cgrid_3d_rl.o exch2_uv_cgrid_3d_rs.o exch2_uv_dgrid_3d_r4.o exch2_uv_dgrid_3d_r8.o exch2_z_3d_r4.o exch2_z_3d_r8.o exch2_z_3d_rl.o exch2_z_3d_rs.o w2_cumulsum_z_tile.o w2_e2setup.o w2_eeboot.o w2_map_procs.o w2_print_comm_sequence.o w2_print_e2setup.o w2_readparms.o w2_set_cs6_facets.o w2_set_f2f_index.o w2_set_gen_facets.o w2_set_map_cumsum.o w2_set_map_tiles.o w2_set_myown_facets.o w2_set_single_facet.o w2_set_tile2tiles.o exf_ad_dump.o exf_adjoint_snapshots_ad.o exf_adjoint_snapshots.o exf_adjoint_snapshots__g.o exf_check_range.o exf_diagnostics_fill.o exf_diagnostics_init.o exf_getffield_start.o exf_getfield_start.o exf_interp_read.o exf_monitor_ad.o exf_zenithangle_table.o gad_advscheme.o gad_check.o gad_diagnostics_init.o gad_diagnostics_state.o gad_init_fixed.o gad_osc_hat_r.o gad_osc_hat_x.o gad_osc_hat_y.o gad_osc_mul_r.o gad_osc_mul_x.o gad_osc_mul_y.o gad_plm_fun.o gad_ppm_adv_r.o gad_ppm_adv_x.o gad_ppm_adv_y.o gad_ppm_flx_r.o gad_ppm_flx_x.o gad_ppm_flx_y.o gad_ppm_fun.o gad_ppm_hat_r.o gad_ppm_hat_x.o gad_ppm_hat_y.o gad_ppm_p3e_r.o gad_ppm_p3e_x.o gad_ppm_p3e_y.o gad_pqm_adv_r.o gad_pqm_adv_x.o gad_pqm_adv_y.o gad_pqm_flx_r.o gad_pqm_flx_x.o gad_pqm_flx_y.o gad_pqm_fun.o gad_pqm_hat_r.o gad_pqm_hat_x.o gad_pqm_hat_y.o gad_pqm_p5e_r.o gad_pqm_p5e_x.o gad_pqm_p5e_y.o gad_write_pickup.o salt_fill.o ggl90_check.o ggl90_diagnostics_init.o ggl90_idemix.o ggl90_output.o ggl90_readparms.o gmredi_calc_eigs.o gmredi_calc_psi_bvp.o gmredi_calc_urms.o gmredi_check.o gmredi_diagnostics_fill.o gmredi_diagnostics_impl.o gmredi_diagnostics_init.o gmredi_init_fixed.o gmredi_k3d.o gmredi_mnc_init.o gmredi_output.o gmredi_readparms.o gmredi_read_pickup.o gmredi_write_pickup.o submeso_calc_psi.o grdchk_check.o grdchk_getadxx.o grdchk_get_obcs_mask.o grdchk_get_position.o grdchk_getxx.o grdchk_init.o grdchk_loc.o grdchk_main.o grdchk_print.o grdchk_readparms.o grdchk_setxx.o grdchk_summary.o mdsio_buffertorl.o mdsio_buffertors.o mdsio_check4file.o mdsio_facef_read.o mdsio_gl.o mdsio_gl_slice.o mdsio_pass_r4torl.o mdsio_pass_r4tors.o mdsio_pass_r8torl.o mdsio_pass_r8tors.o mdsio_rd_rec_rl.o mdsio_rd_rec_rs.o mdsio_read_field.o mdsio_read_meta.o mdsio_read_section.o mdsio_read_tape.o mdsio_readvec_loc.o mdsio_read_whalos.o mdsio_rw_field.o mdsio_rw_slice.o mdsio_seg4torl.o mdsio_seg4tors.o mdsio_seg8torl.o mdsio_seg8tors.o mdsio_segxtorx_2d.o mdsio_write_field.o mdsio_writelocal.o mdsio_write_section.o mdsio_write_tape.o mdsio_writevec_loc.o mdsio_write_whalos.o mdsio_wr_metafiles.o mdsio_wr_rec_rl.o mdsio_wr_rec_rs.o mon_advcfl.o mon_advcflw2.o mon_advcflw.o mon_calc_advcfl.o mon_calc_stats_rl.o mon_calc_stats_rs.o mon_init.o monitor_ad.o monitor.o monitor_g.o mon_ke.o mon_out.o mon_printstats_rl.o mon_printstats_rs.o mon_set_iounit.o mon_set_pref.o mon_solution.o mon_stats_latbnd_rl.o mon_stats_rl.o mon_stats_rs.o mon_surfcor.o mon_vort3.o mon_writestats_rl.o mon_writestats_rs.o active_file_control_profiles.o active_file_profiles_ad.o active_file_profiles.o active_file_profiles_g.o profiles_findunit.o profiles_ini_io.o profiles_init_ncfile.o profiles_readparms.o profiles_readvector.o get_write_global_fld.o read_glvec_rl.o read_glvec_rs.o read_mflds.o rw_get_suffix.o set_write_global_fld.o write_fld_3d_rl.o write_fld_3d_rs.o write_fld_s3d_rl.o write_fld_s3d_rs.o write_fld_xy_rl.o write_fld_xy_rs.o write_fld_xyz_rl.o write_fld_xyz_rs.o write_glvec_rl.o write_glvec_rs.o write_local_rl.o write_local_rs.o write_rec.o salt_plume_check.o salt_plume_diagnostics_init.o salt_plume_init_fixed.o salt_plume_mnc_init.o salt_plume_readparms.o sbo_calc.o sbo_check.o sbo_output.o sbo_readparms.o sbo_rho.o seaice_ad_dump.o seaice_calc_lhs.o seaice_calc_residual.o seaice_calc_rhs.o seaice_check.o seaice_cost_init_fixed.o seaice_cost_weights.o seaice_diagnostics_init.o seaice_do_ridging.o seaice_fgmres.o seaice_init_fixed.o seaice_itd_pickup.o seaice_itd_redist.o seaice_itd_remap.o seaice_itd_sum.o seaice_jacvec.o seaice_jfnk.o seaice_krylov.o seaice_mnc_init.o seaice_monitor_ad.o seaice_monitor.o seaice_obcs_output.o seaice_preconditioner.o seaice_prepare_ridging.o seaice_readparms.o seaice_summary.o seaice_turnoff_io.o smooth_filtervar2d.o smooth_filtervar3d.o smooth_init2d.o smooth_init3d.o smooth_init_fixed.o smooth_readparms.o timeave_init_fixed.o mom_calc_3d_strain.o mom_calc_smag_3d.o mom_diagnostics_init.o mom_init_fixed.o mom_u_botdrag_impl.o mom_u_implicit_r.o mom_uv_smag_3d.o mom_v_botdrag_impl.o mom_v_implicit_r.o mom_w_coriolis_nh.o mom_w_metric_nh.o mom_w_sidedrag.o mom_w_smag_3d.o all_proc_die.o bar2.o bar_check.o barrier.o check_threads.o comm_stats.o cumulsum_z_tile.o diff_phase_multiple.o eeboot.o eeboot_minimal.o eedata_example.o eedie.o eeintro_msg.o eeset_parms.o eewrite_eeenv.o exch0_r4.o exch0_r8.o exch0_rl.o exch0_rs.o exch1_bg_r4_cube.o exch1_bg_r8_cube.o exch1_bg_rl_cube.o exch1_bg_rs_cube.o exch1_r4_cube.o exch1_r4.o exch1_r8_cube.o exch1_r8.o exch1_rl_ad.o exch1_rl_cube_ad.o exch1_rl_cube.o exch1_rl.o exch1_rs_ad.o exch1_rs_cube_ad.o exch1_rs_cube.o exch1_rs.o exch1_uv_r4_cube.o exch1_uv_r8_cube.o exch1_uv_rl_cube.o exch1_uv_rs_cube.o exch1_z_r4_cube.o exch1_z_r8_cube.o exch1_z_rl_cube.o exch1_z_rs_cube.o exch_3d_r4.o exch_3d_r8.o exch_cycle_ebl.o exch_init.o exch_r4_recv_get_x.o exch_r4_recv_get_y.o exch_r4_send_put_x.o exch_r4_send_put_y.o exch_r8_recv_get_x.o exch_r8_recv_get_y.o exch_r8_send_put_x.o exch_r8_send_put_y.o exch_rl_recv_get_x.o exch_rl_recv_get_y.o exch_rl_send_put_x.o exch_rl_send_put_y.o exch_rs_recv_get_x.o exch_rs_recv_get_y.o exch_rs_send_put_x.o exch_rs_send_put_y.o exch_s3d_r4.o exch_s3d_r8.o exch_s3d_rl.o exch_s3d_rs.o exch_sm_3d_r4.o exch_sm_3d_r8.o exch_sm_3d_rs.o exch_uv_3d_r4.o exch_uv_3d_r8.o exch_uv_agrid_3d_r4.o exch_uv_agrid_3d_r8.o exch_uv_bgrid_3d_r4.o exch_uv_bgrid_3d_r8.o exch_uv_bgrid_3d_rl.o exch_uv_bgrid_3d_rs.o exch_uv_dgrid_3d_r4.o exch_uv_dgrid_3d_r8.o exch_uv_xy_r4.o exch_uv_xy_r8.o exch_uv_xyz_r4.o exch_uv_xyz_r8.o exch_xy_r4.o exch_xy_r8.o exch_xyz_r4.o exch_xyz_r8.o exch_z_3d_r4.o exch_z_3d_r8.o exch_z_3d_rl.o exch_z_3d_rs.o fool_the_compiler.o gather_2d_r4.o gather_2d_r8.o gather_2d_wh_r4.o gather_2d_wh_r8.o gather_vec_r4.o gather_vec_r8.o gather_xz.o gather_yz.o global_max.o global_sum.o global_sum_singlecpu.o global_sum_tile.o global_vec_sum.o gsum.o ini_communication_patterns.o ini_procs.o ini_threading_environment.o main.o master_cpu_io.o master_cpu_thread.o mds_byteswapi4.o mds_byteswapr4.o mds_byteswapr8.o mdsfindunit.o mds_flush.o mds_reclen.o memsync.o nml_change_syntax.o nml_set_terminator.o open_copy_data_file.o print.o reset_halo.o scatter_2d_r4.o scatter_2d_r8.o scatter_2d_wh_r4.o scatter_2d_wh_r8.o scatter_vec_r4.o scatter_vec_r8.o scatter_xz.o scatter_yz.o stop_if_error.o timers.o utils.o write_utils.o add_walls2masks.o calc_eddy_stress.o calc_grid_angles.o calc_gw.o calc_oce_mxlayer.o cg2d_ex0.o cg2d.o cg2d_sr.o cg3d_ex0.o cg3d.o check_pickup.o config_check.o config_summary.o diags_oceanic_surf_flux.o do_statevars_diags.o do_statevars_tave.o do_write_pickup.o external_forcing.o find_hyd_press_1d.o gsw_teos10.o ini_cartesian_grid.o ini_cg3d.o ini_cori.o ini_curvilinear_grid.o ini_cylinder_grid.o ini_eos.o ini_global_domain.o ini_grid.o ini_linear_phisurf.o ini_local_grid.o ini_masks_etc.o ini_mnc_vars.o ini_model_io.o ini_nh_vars.o ini_parms.o ini_sigma_hfac.o ini_spherical_polar_grid.o initialise_fixed.o ini_vertical_grid.o load_ref_files.o packages_boot.o packages_check.o packages_error_msg.o packages_init_fixed.o packages_print_msg.o packages_readparms.o packages_unused_msg.o packages_write_pickup.o plot_field.o port_rand.o post_cg3d.o pre_cg3d.o rotate_spherical_polar_grid.o set_defaults.o set_grid_factors.o set_parms.o set_ref_state.o solve_uv_tridiago.o the_model_main.o timestep_wvel.o tracers_iigw_correction.o turnoff_model_io.o write_grid.o write_pickup.o write_state.o gsl_ieee_env.o ptwrapper.o setdir.o setrlstk.o sigreg.o tim.o timer_stats.o -L/opt/cray/pe/netcdf-hdf5parallel/4.7.4.3/crayclang/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/crayclang/9.1/include/lib /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: failed to convert GOTPCREL relocation; relink with --no-relax make[1]: [Makefile:2773: mitgcmuv_ad] Error 1 make[1]: Leaving directory '/home2/home/n01/n01/emmomp/MITgcm/ECCO-v4-Configurations/ECCOv4 Release 4/build_ff_ad' make: [Makefile:2750: ad_exe_target] Error 2

emmomp commented 2 years ago

Apologies, a quick google revealed this is a problem with large executables.

I resolved this by using the gnu compiler instead of cray, and using the flags: FFLAGS="$FFLAGS -mcmodel=medium" CFLAGS="$CFLAGS -mcmodel=medium"