`use_concat` option not fully supported when False

[trjr]

The use_concat option acts in the deformation function to save deformation fields in a single concatenated form (with i and j dimensions; True) or to save these as individual deformation_ij fields (False). The option also dictates the form of the returned value: an xarray DataArray in the former, and a dictionary of 9 xarray DataArrays in the latter.

When False, the default in earlier versions of Subfilter, though, follow-on routines that derive further quantities from the deformation will fail, as they expect the concatenated field form for processing. These include:

• filtered_deformation, which immediately tries to rechunk the result (and will not directly act on the dictionary form)
• shear, which immediately acts to select fields based on xarray i and j dimensions.

These routines can be amended to perform different behaviour based on the type of the field or the use_concat flag. Otherwise, we might consider removing the option to have these as individual fields. The former is ultimately more flexible, but I'm not sure whether it's necessary.

[ReadingClouds]

Dear Todd

My feeling is that the easiest thing to do is lose the alternative storage option (at least until circumstances arise where we are forced to re-instigate it). Have you had any instances where one worked but the other didn't?

BW

Peter

Prof. Peter Clark Department of Meteorology University of Reading Tel: +44(0) 118 378 7908<tel:+44%20118%20378%207908> email: @.**@.>

From: Todd Jones @.> Sent: 17 February 2022 09:15 To: ReadingClouds/Subfilter @.> Cc: Subscribed @.***> Subject: [ReadingClouds/Subfilter] use_concat option not fully supported when False (Issue #17)

The use_concat option acts in the deformation function to save deformation fields in a single concatenated form (with i and j dimensions; True) or to save these as individual deformation_ij fields (False). The option also dictates the form of the returned value: an xarray DataArray in the former, and a dictionary of 9 xarray DataArrays in the latter.

When False, the default in earlier versions of Subfilter, though, follow-on routines that derive further quantities from the deformation will fail, as they expect the concatenated field form for processing. These include:

• filtered_deformation, which immediately tries to rechunk the result (and will not directly act on the dictionary form)
• shear, which immediately acts to select fields based on xarray i and j dimensions.

These routines can be amended to perform different behaviour based on the type of the field or the use_concat flag. Otherwise, we might consider removing the option to have these as individual fields. The former is ultimately more flexible, but I'm not sure whether it's necessary.

- Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FReadingClouds%2FSubfilter%2Fissues%2F17&data=04%7C01%7Cp.clark%40reading.ac.uk%7Cf1aca5191aa4499ab4c908d9f1f5f82f%7C4ffa3bc4ecfc48c09080f5e43ff90e5f%7C0%7C0%7C637806860996149986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=eEjd8UFv2%2FizsCpvTgex0507%2FYSHzNjG5h%2BMtz15cas%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALQXWELVFYX2ON5JR6Z5543U3S4BDANCNFSM5OUB4UFQ&data=04%7C01%7Cp.clark%40reading.ac.uk%7Cf1aca5191aa4499ab4c908d9f1f5f82f%7C4ffa3bc4ecfc48c09080f5e43ff90e5f%7C0%7C0%7C637806860996149986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ffhiwWjDcub67w40w0XaUy98KwmbXXmPnAE6iptx38c%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cp.clark%40reading.ac.uk%7Cf1aca5191aa4499ab4c908d9f1f5f82f%7C4ffa3bc4ecfc48c09080f5e43ff90e5f%7C0%7C0%7C637806860996149986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=mwCMR4xFuigdy2ZxlBYjNicdGEc8o3YlzID8ZukS6BQ%3D&reserved=0 or Androidhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cp.clark%40reading.ac.uk%7Cf1aca5191aa4499ab4c908d9f1f5f82f%7C4ffa3bc4ecfc48c09080f5e43ff90e5f%7C0%7C0%7C637806860996149986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8k3ZLNV4CozxJVDIV6L694NA3Becm7yfA96K15rJFrc%3D&reserved=0. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>

[trjr]

Both options work on their own for calculation and saving to file. As a difference, one might notice that the concatenated version can take longer to form and write.

The issue is only that they return different deformation variable structures that cannot be directly used by routines that expect the concatenated form.

I agree that the simplest option is to remove the granular component variable writes as an option.