Investigate Data Format Description Language (DFDL)

wader / fq

jq for binary formats - tool, language and decoders for working with binary and text formats

Other

9.57k stars 219 forks source link

Investigate Data Format Description Language (DFDL) #822

Open nichtich opened 7 months ago

nichtich commented 7 months ago

This is a rather broad issue, not a specific feature request, as I did not find any mention of [Data Format Description Language (DFDL)]() in this repository.

I have not worked with DFDL but as far as I understand it, given a DFDL schema (see examples here), a DFDL processor should be able to parse data in this format. To allow fq parsing arbitrary formats with known DFDL schema it needs

either an implementation of a DFDL processor in Go
or a binding to an existing DFDL processor (probably Apache Daffodil) from fq
or a compiler to create Go code from a DFDL schema

Sure, support of DFDL is far from trivial and I don't know if its worth the effort, given the current popularity DFDL, but in any case fq and DFDL are related enough to better join some efforts in the long run.

wader commented 7 months ago

Hey, looks interesting and it reminds me of kaitai. I've spent some time on a kaitai interpreter prototype for fq which looks promising but there is quite a lot of work left on it, but i think DFDL and kaitai would be implemented in similar ways.

The main difficulties i can see is:

Implement the DFDL expression language
Is the java regular expression dialect compatible enough or translatable to golang stdlib regex?
How complex is the type system? for kaitai i run into some difficulties understanding how types are resolved etc

Is this something you would like to work on? i'm thinking maybe i should refactor out some things from my kaitai branch that can probably be shared.

wader commented 7 months ago

BTW there is a related issue https://github.com/wader/fq/issues/627 about runtime decoding

nichtich commented 7 months ago

Is this something you would like to work on?

No, just wanted to create a link betwee DFDL and fq but missed the obvious link in README.md as I only searched for the acronym "DFDL". Seems like nobody has mentioned fq and neither Kaitai in the Daffodil issue tracker and mailing lists.

If you happen to implement Kaitai struct anyway, mapping DFDL to Kaitai seems a better approach because such mapping could also be used independently from fq (https://github.com/kaitai-io/kaitai_struct/issues/441). I guess it won't support all features of DFDL but enough to get the communities in touch.

wader commented 7 months ago

If you happen to implement Kaitai struct anyway, mapping DFDL to Kaitai seems a better approach because such mapping could also be used independently from fq (kaitai-io/kaitai_struct#441). I guess it won't support all features of DFDL but enough to get the communities in touch.

By mapping you mean add DFDL output support to the kaitai projects? yeap that sounds resonable