storing DAQPartition objects in test data directory

[andreh12]

We could persist the DAQPartition objects for the hardware configuration keys needed in FlashlistDispatcherIT instead of reading them from the hardware database. This would greatly speed up the test (but it may be against the spirit of an integration test).

[gladky]

I was considering to do this in the past for the same reason. Unfortunately I didn't find time to do this but I think is worth implementing to avoid hardware access in tests. We could persist it in similar way as snapshots to solve the circular relations issue.

[andreh12]

We discussed two possible approaches to this problem:

1. write the data obtained from the hardware database used by our integration tests into a sqlite file which can then be stored in the github repository
2. serialize the DAQPartition using Jackson and store them as zipped json files in the repository.

---

2. Looks apriori easier to implement. However it turns out that there are some circular dependencies which need to be properly dealt with (otherwise Jackson generates a . And since DAQPartition is defined outside the DAQAggregator this is not straightforward.

Out of the box, Jackson throws an exception complaining about infinite recursion:

Infinite recursion (StackOverflowError) (through reference chain: rcms.utilities.hwcfg.eq.NIC["dnsnames"]->
java.util.TreeSet[0]->
rcms.utilities.hwcfg.eq.DNSName["nic"]->
rcms.utilities.hwcfg.eq.NIC["dnsnames"]->
java.util.TreeSet[0]->
(many repetitions of the above)
)

[gladky]

This problem was encountered for serialising snapshots as well. The solution is to stop the serialisation of objects that have been already serialised. One way to do this is to use generated ids to refer to them. This should solve the circular dependency:

@JsonIdentityInfo(
        generator = ObjectIdGenerators.IntSequenceGenerator.class,
        property = "@id")
abstract class ObjectMixIn { }

This will generate integer id for each serialised object.

ObjectMapper objectMapper = new ObjectMapper();
objectMapper.addMixIn(Object.class, ObjectMixIn.class);
System.out.println(objectMapper.writeValueAsString(rootObjectToSerialize));

Here you register the mixin to your mapper targeting Object class (it will be enabled for all classes) and this should fix the circular dependencies replacing some occurrences with integer id. Deserialisation would work using the same mechanism.

[andreh12]

Thanks for the hint @gladky ! I'll give it a try.

I looked at the source code of rcms.utilities.hwcfg.eq.NIC and rcms.utilities.hwcfg.eq.DNSName and indeed NIC has a field _dnsNames of type Set<DNSName> and class DNSName has a field _nic of type NIC pointing back to the NIC.

[andreh12]

In the end I failed to get this to work with Jackson, despite attempts with using custom serializers, declaring parent/child relationships etc. For some reason jackson always ends up with messages that it expected a number key for a map when it got a string like fmm_ etc.

It's also difficult to debug this since many objects point to many others in the RCMS hardware database model and serializing e.g. a single FED leads to the entire partition being serialized if one does not follow the path to the pointer back to the partition and sets it to null by hand in tests.

The solution with sqlite on the other hand is more lightweight: it is straightforward to copy the contents of SQL tables without hardwiring the column structure into the code. The only caveat was that the hardware database has one query with a vendor-specific SQL extension which is not supported by sqlite was easily intercepted and rewritten.

Pull request #108 implements the sqlite solution.