search

oracle / coherence

Oracle Coherence Community Edition
https://coherence.community
Universal Permissive License v1.0
426 stars 70 forks source link

TDE and CDB files #38

Closed leolegenie closed 2 years ago

leolegenie commented 3 years ago

Type of question

I would like to know more about the background of the TDE toolchain and the CDB files which seem to be used as basis for generating large parts of the Coherence code.

Question

What did you do?

I wanted to do some experiments by tweaking the method public QueryResult query(Filter filter, int nQueryType, PartitionSet partMask) in com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage but struggled to find the Java source code for this.

It seems that the code is generated out of tde/core-net/3.0/cdb/Component/Util/Daemon/QueueProcessor/Service/Grid/PartitionedService/PartitionedCache.cdb (and a lot of other code too analogously)?

I found the tde utility (Tangsol Desktop) under tools/bin/tde which allows to open those cdb files and look at them/modify them. With a bit of try and error I managed to modify the method I was interested in and by re-running the toolchain get the result I wanted to in the first place, but to be honest, half of the time I did not understand what I was doing.

So my questions are:

  1. What are those cdb files and the tde tool and what is the general idea behind it? Is this something tangosol-properitary or part of/based on some other tooling?
  2. Is there some documentation on how it all works in more details (e. g. what do all the options mean in tde, how as a developer you are supposed to use the tool, where would you do changes to code if you want to introduce new functionality, etc.)?
  3. Are there plans to move away from this toolchain to something, let's say, more intuitive to the regular developer or is the current situation that tde/cdb is just too tightly integrated into the Coherence DNA to make it worth the effort?

Thanks, regards, Leo

What did you expect to see? I would like to see documentation about TDE and the CDB format.

What did you see instead? Under which circumstances? I couldn't find any documentation.

Environment N/a

Additional context N/a

harveyraja commented 3 years ago

Hi @leolegenie,

Here are some answers to the specific Qs you called out:

  1. a cdb file is a binary format that serializes the entire definition of a Component including traits, behaviors, sub components, properties, etc. These cdb files are viewed, edited, created and deleted via TDE (Tangosol Development Environment). It is proprietary and we put v.little investment in the tooling and compiler.
  2. Our documentation is pretty thin - non-existent in this area other than what is more like a cheat sheet here. We will be developing some material to help both internal and external audiences develop with components easier as it truly is the core of the product.
  3. We decided against a sweeping movement to migrate all components to some other JVM-based language and instead are designing new features to be primarily developed in java and provide integration points with the various components they need to integrate with, where possible. Good examples of this are persistence and our transport layer (message bus).

Happy to keep this issue open until we make good on our promise to provide better docs.

leolegenie commented 3 years ago

Thanks for the info, I created some notes on how to get started with the TDE tool here, just in case.

aseovic commented 2 years ago

Just wanted to comment on this in case anyone else is learning to work with TDE and edit CDB files.

We now generate compileable Java source in addition to compiled classes during TDE compilation, as part of coherence-core-components module build. We actually compile them as well, to make sure the source continues to compile as we make changes to CDBs it is created from, and you can also choose to compile and package the classes created from the generated sources by specifying -Ptde.sources when doing Maven build (instead of TDE-generated classes, which is the default, or if -Ptde.classes option is specified explicitly).

This allows you mark coherence-core-components/target/generated-sources as "generated sources" directory in IntelliJ and read, debug and even try things out in IntelliJ by making changes to the generated sources and running tests against them directly. Of course, any changes you make have to be applied to the right CDB files in order to be made permanent, at least for the time being.

leolegenie commented 2 years ago

Thanks, very helpful, particularly for debugging!