Feature: execute graphs with beam-dimap input

[jonathom]

Hi all, I'm using pyroSAR to create, group, and then execute a graph via SNAP, the last of which via gpt from pyroSAR.snap.auxil. It seems to me that this operation fails if I want to use .dim BEAM-DIMAP formatted intermediate products from SNAP as inputs to my workflow (It's raising RuntimeError: data format not supported). The same graph also fails in the SNAP graph tool, but works fine if I execute it via gpt in the command line.

From what I've seen in the code, I'm guessing that I can only execute graphs with pyroSAR that contain compressed formats (e.g. SAFE .zips) in their READ nodes?

I also wanted to ask if this might be a feature in which more people might be interested / that would be easy to implement?

[johntruckenbrodt]

Hi @jonathom. I agree, this would be a great addition. So far Sentinel-1 scenes can only be processed if they are either in the original zip or unpacked in a .SAFE folder. The function pyroSAR.drivers.identify is responsible for this. It tries to read the scene with the available SAR scene drivers (like pyroSAR.drivers.SAFE), which all inherit from the class pyroSAR.drivers.ID.
In order to make all this work it would need a new class for the BEAM-DIMAP format that makes use of this mechanism. For starters not all methods that are suggested by the ID class have to be implemented. Then it would be easy for the gpt function to identify the source product and execute the workflow accordingly.

@konstantinschellenberg you have written some class right? Would you be willing to contribute it?

[sirbastiano]

Hi johntruckenbrodt. What about other platforms' products (e.g. Cosmo-Skymed)? Are they supported?

[johntruckenbrodt]

Hi @UninaLabs-EO. The supported formats are listed here. Cosmo-Skymed is not among them. You are more than welcome to contribute something :wink:.

[sirbastiano]

Yeah, for sure. From my perspective, and correct me if I am wrong, what should be changed is the node=read. I have tested xml graphs on GPT and they work with Cosmo. So, basically, they should work also for PyroSAR. Could you suggest me where to start?

[sirbastiano]

@johntruckenbrodt, for what I have seen, the parameters should contain this class: parameters class="com.bc.ceres.binding.dom.XppDomElement"> param param /parameters>

[johntruckenbrodt]

Great! The most solid way would be to add a dedicated class to the pyroSAR.drivers module, for example named CSK. This is needed to properly manage the scene in pyroSAR. What you describe is "just" the parametrization of a SNAP workflow. This should easily be possible already with the SNAP API. There you can find an example of how to create a workflow and save it to an XML file. It can then be executed with the function pyroSAR.snap.auxil.execute. The function pyroSAR.snap.auxil.gpt would not work in this case because it requires the dedicated class to identify the scene and read metadata.

[sirbastiano]

Yeah, you are perfectly right. I manage to handle the workflow modifying the gpt function with a skip on the scene if the product is CSK. However, reading a CSK scene with a custom class would be the best. I'll try and let you know. Thanks a lot John!

[johntruckenbrodt]

Sounds great @UninaLabs-EO. This would be a great addition! Feel free to reach out via a dedicated issue or pull request for further discussion.

[pbrotoisworo]

Hi @johntruckenbrodt , I just started using pyroSAR and I came across this issue myself when trying to load .dim files from SNAP.

I'm curious if I can contribute something here since I have a few repos related to BEAM-DIMAP parsing. Just to clarify, what is self.pattern in each sensor class? Is that supposed to point to the metadata locations in the ZIP file? In the case of BEAM-DIMAP, the .dim file itself contains the metadata. Would I just leave that empty?

[pbrotoisworo]

Hello, just a quick update. Managed to get things working by just creating this basic class and inserting it into drivers.py. This worked for my Sentinel-1 interferogram test data which I just ran a simple debursting operation on.

The main issues I encountered are that existing methods expect tar or zip files but a .dim is essentially an XML file. I would just create BEAM-DIMAP related methods under BEAM_DIMAP to parse the required metadata.

Note: This is just an initial proof of concept. Please excuse all the NoneTypes. 😄

class BEAM_DIMAP(ID):

    def __init__(self, scene):

        if not scene.lower().endswith('.dim'):
            raise RuntimeError('Scene format is not BEAM-DIMAP')

        root = ET.parse(scene).getroot()
        self.meta = dict()
        self.meta['acquisition_mode'] = root.find('.//MDATTR[@name="ACQUISITION_MODE"]').text
        self.meta['IPF_version'] = root.find('.//MDATTR[@name="Processing_system_identifier"]').text
        self.meta['sensor'] = root.find('.//MDATTR[@name="MISSION"]').text
        self.meta['orbit'] = root.find('.//MDATTR[@name="PASS"]').text
        self.meta['polarizations'] = list(set([x.text for x in root.findall('.//MDATTR[@desc="Polarization"]') if '-' not in x.text]))
        self.meta['start'] = None
        self.meta['stop'] = None
        self.meta['spacing'] = None
        self.meta['samples'] = None
        self.meta['lines'] = None
        self.meta['orbitNumber_abs'] = root.find('.//MDATTR[@name="ABS_ORBIT"]').text
        self.meta['orbitNumber_rel'] = root.find('.//MDATTR[@name="REL_ORBIT"]').text
        self.meta['cycleNumber'] = root.find('.//MDATTR[@name="cycleNumber"]').text
        self.meta['frameNumber'] = None
        self.meta['product'] = root.find('.//DATASET_COMMENTS').text
        self.meta['projection'] = None

        super(BEAM_DIMAP, self).__init__(self.meta)

[johntruckenbrodt]

Hi @pbrotoisworo. Thanks a lot for looking into this! self.pattern is just for identifying the file name unique to the format. This could either be the name of the zip/tar or a file/folder inside. For example, for SAFE it it the .SAFE folder inside the zip archive. In the BEAM-DIMAP case the simple check whether the file has the extension .dim that you have implemented should be enough. In general the class ID is intended to serve as a blueprint for other classes. It contains all methods that are required by a new class somewhere in pyroSAR. Some need to be overridden in the respective class implementation. For example, there is a method ID.scanMetadata that would need to be overridden by BEAM_DIMAP.scanMetadata. Another important one is ID.getCorners. Furthermore, the class needs to define all of the attributes listed in __LOCAL__. I think if you complete the metadata reading into self.meta, move it all into a dedicated method scanMetadata, and define getCorners you should be good to go.

[pbrotoisworo]

Thanks for the pointers. I'll give this a shot and I'll update this thread if I have any other concerns.