search

MetOffice / opsinputs

JEDI library generating VarObs and Cx files
BSD 3-Clause "New" or "Revised" License
4 stars 1 forks source link

Update varobs and cx writers for visibility #221

Closed ReubenHill closed 2 months ago

ReubenHill commented 2 months ago

The changes made in this PR are necessary for the assimilation of visibility by VAR. There are three key changes:

  1. The surface namelist has been updated to include model fields for visibility, surface total water (qt2) and aerosol (specifically with stash code 90, called "Total Aerosol (for Vis)").
  2. The cx writer has been updated to allow surface total water and aerosol to be output. These are used by VAR, alongside other model variables, to calculate background visibility.
  3. The varobs writer will now write ObsValue/horizonalVisibility values to VarField_logvis.

Updating the cx writer for outputting aerosol was non-trivial:

  1. There are multiple aerosol stash codes - 90 is the correct one to use for the calculation of background visibility in VAR.
  2. The field is defined over all 70 levels, but OPS only outputs the level closest to the surface: see the comments in src/opsinputs/opsinputs_fill_mod.F90 for details. This required special case code.

@adammaycock this was the code you were helping me to debug last month: the issue was an incorrect stash code causing the necessary select case to be skipped over.

@ss421 I have now confirmed that I need the following for VAR to run, so please don't exclude them from FieldNameCompare.xlsx:

NOTE: the ukv surface KGO will need to be remade once this is merged.

ReubenHill commented 2 months ago

Very nice! Does this require any modifications to sith KGO output?

I don't think so: don't KGOs only use the NetCDF output from the jopa background task? This ought not to effect that.

ctgh commented 2 months ago

I believe they check ODB, Varobs and CX output. It might be worth running it just to be on the safe side.

ReubenHill commented 2 months ago

I believe they check ODB, Varobs and CX output. It might be worth running it just to be on the safe side.

Yes you are correct, the kgos will need to be remade. Unless it's very straightforward, it's probably only worth doing once the jjdocs PR is ready since that will also need new kgos.

ReubenHill commented 2 months ago

OFFICIAL

I guess we don't have a UKV surface KGO set up then. Please merge if you are happy.

OFFICIAL

From: Matt Shin @.> Sent: 16 August 2024 4:16 PM To: MetOffice/opsinputs @.> Cc: Reuben Nixon-Hill @.>; Assign @.> Subject: Re: [MetOffice/opsinputs] Update varobs and cx writers for visibility (PR #221)

This email was received from an external source. Always check sender details, links & attachments.

@matthewrmshin approved this pull request.

Ran OK in my environment. No failures detected for mo-bundle main build + ctest, and sith/malak kgo runs.

— Reply to this email directly, view it on GitHubhttps://github.com/MetOffice/opsinputs/pull/221#pullrequestreview-2242953616, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB4I73KNSWJ5T3DO565QRXDZRYJWVAVCNFSM6AAAAABMSNKOV6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDENBSHE2TGNRRGY. You are receiving this because you were assigned.Message ID: @.***>

matthewrmshin commented 2 months ago

OFFICIAL

Hi Reuben,

I just realised my typing was cut off halfway. I actually meant that there were no unexpected KGO issues. I've now amended my comment in the PR. Sorry for the confusion.

Best, Matt

Dr Matt Shin Expert Scientific Software Engineer Met Office FitzRoy Road Exeter EX1 3PB United Kingdom Tel: +44 (0)330135 1724 @.*** http://www.metoffice.gov.uk/

OFFICIAL

From: Reuben W. Nixon-Hill @.> Sent: 16 August 2024 16:26 To: MetOffice/opsinputs @.> Cc: Matthew Shin @.>; Mention @.> Subject: Re: [MetOffice/opsinputs] Update varobs and cx writers for visibility (PR #221)

This email was received from an external source. Always check sender details, links & attachments.

OFFICIAL

I guess we don't have a UKV surface KGO set up then. Please merge if you are happy.

OFFICIAL

From: Matt Shin @.> Sent: 16 August 2024 4:16 PM To: MetOffice/opsinputs @.> Cc: Reuben Nixon-Hill @.>; Assign @.> Subject: Re: [MetOffice/opsinputs] Update varobs and cx writers for visibility (PR #221)

This email was received from an external source. Always check sender details, links & attachments.

@matthewrmshin approved this pull request.

Ran OK in my environment. No failures detected for mo-bundle main build + ctest, and sith/malak kgo runs.

— Reply to this email directly, view it on GitHubhttps://github.com/MetOffice/opsinputs/pull/221#pullrequestreview-2242953616, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB4I73KNSWJ5T3DO565QRXDZRYJWVAVCNFSM6AAAAABMSNKOV6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDENBSHE2TGNRRGY. You are receiving this because you were assigned.Message ID: @.***>

— Reply to this email directly, view it on GitHubhttps://github.com/MetOffice/opsinputs/pull/221#issuecomment-2293714357, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAG52RI6HYFYIO53FAAZ3NLZRYK2NAVCNFSM6AAAAABMSNKOV6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJTG4YTIMZVG4. You are receiving this because you were mentioned.Message ID: @.***>

matthewrmshin commented 2 months ago

This change appears to require KGO changes for various Surface.* files for sith kgo_glu and kgo_ukv as well as malak kgo_glu_jopa_jada and kgo_glu_jopa_var - so not just the UKV ones. Is this expected?

ReubenHill commented 2 months ago

This change appears to require KGO changes for various Surface.* files for sith kgo_glu and kgo_ukv as well as malak kgo_glu_jopa_jada and kgo_glu_jopa_var - so not just the UKV ones. Is this expected?

The varobs writer now outputs visibility - since these are in the list of observed variables for both global and ukv yamls, I therefore expect both sets of varobs files to change. The changes here should not effect global surface cx files, though I can imagine that a change to a varobs file might somehow necessitate a change to a cx file. What exact differences do you see?

matthewrmshin commented 2 months ago

Differences reported for these files:

malak:

sith:

See outputs of these suites of mine: http://fcm1/cylc-review/suites?user=frsn&names=*oi221-kgo_glu*+*oi221-kgo_ukv

ReubenHill commented 2 months ago

Differences reported for these files:

malak:

  • kgo_glu_jopa_jada:

    • glu_jopa_varobs/Surface.varobs

  • kgo_glu_jopa_var:

    • glu_jopa_varobs/Surface.varobs

sith:

  • kgo_glu:

    • glu_jopa_varobs/Surface.varobs
    • glu_jopa_varobs_screen/Surface.varobs

  • kgo_ukv:

    • ukv_jopa_var_cx/Surface.cx
    • ukv_jopa_varobs/Surface.varobs

See outputs of these suites of mine: http://fcm1/cylc-review/suites?user=frsn&names=*oi221-kgo_glu*+*oi221-kgo_ukv

These are as expected. Whilst the global varobs files now contain visibility (as the OPS ones always have), the global JOPA VAR task job.stats outputs before and after are identical - i.e. the presence of visibility in the global surface JOPA output has no impact on VAR. So please go ahead and recreate the KGOs.