use of collect / count and sort triple commands

[guenterh]

Problem

• for some cases we are using the sort-triples and collect-triples commands to process large amounts of data. E.g. to analyze bibliographic data or for transformations of work entities
• https://github.com/linked-swissbib/mfWorkflows/blob/master/src/main/resources/transformation/work/work.flux#L16
• https://github.com/linked-swissbib/mfWorkflows/blob/master/src/main/resources/statistics/item.flux#L11
• https://github.com/linked-swissbib/mfWorkflows/blob/master/src/main/resources/transformation/jsonld/transformation.flux#L48

We get in trouble because of insufficient memory and had to split the data in smaller sets for processing which is not only cumbersome but also gives wrong results because analysis has to be done on the complete data set.

Question

I have seen there is a mechanism triggered by a flag called 'memorylow' https://github.com/culturegraph/metafacture-core/blob/master/src/main/java/org/culturegraph/mf/stream/pipe/sort/AbstractTripleSort.java#L99 which makes it possible to swap triples to the file system as temporary store

• is this mechanism used by any other? Are there any experiences with it?
• As far as I have seen is the memorylow flag only accessible via Java (not Flux) because there is no setter-method. Did I miss something?

  own steps so far

• I created a test client app where I used the count and sort commands together with the memoryflag mechanism.
• the memorylow signature requires long parameters which are never used
• the current process function evaluates the memoryflag but this mechanisms can't work IMHO because calling 'process' for the first time the buffer is always empty and memorylow is set to false which deactivates the removal to external storage mechanism immediately.
• I changed this a little bit just to not switch off the removal machanisms and to see how it works. Of course there should be a more sophisticated implementation
• But: the sort and count mechanisms work as expected: The tempFiles collections isn't empty and the temporarily stored triples are being read using a SortedTripleFileFacade
• for me it seems the mechanism was principally designed but not finished. But: I guess something like this is necessary to analyse large data sets at the latest e.g. to cluster the whole content of all the German library networks as it was done by @mgeipel and @cboehme
  Christoph: could you provide more background information? - Thanks a lot
• other possibilities could be the following commands:
  • write-triples org.culturegraph.mf.stream.sink.TripleWriter
  • write-triple-objects org.culturegraph.mf.stream.sink.TripleObjectWriter
  • reorder-triple org.culturegraph.mf.stream.pipe.TripleReorder which I haven't tried out by myself so far. Are there any practical examples how to use them? This would be really nice. It the spirit of our get together meeting to share more examples?

Thanks for any hints - Günter!

[cboehme]

Hi Günter,

the mechanism for memory management in the TripleSort and TripleCollect is fully implemented. The memoryLow() method is not called by the user but it is invoked from the MemoryWarningSystem every time more than 80% of memory available in the jvm is used. The next time the process() is called it checks whether memoryLow() was invoked and if it was then the current list of triples is saved to disk. This mechanism works automatically and does not need to be configured explicitly.

The idea behind this logic was to enable the user to sort arbitrarily large data sets without having to tinker with the memory settings of the jvm. However, it has turned out that the mechanism is not as tinker-free as we had hoped.

The main issue seems to be that the AbstractTripleSort does not immediately free its memory when the 80% threshold is reached but only once the process() method is called. If the remaining 20% of memory are not enough to fulfill all memory allocations before process() is called the next time then the JVM will throw an OutOfMemoryException. We have encountered this problem as well and have not yet found a good solution for it. What helps as a workaround is to increase the JVM memory so that once the 80% threshold is reached, more memory in absolute numbers is still available.

If you have an idea for a better implementation of the automatic memory management we are glad to change the current implementation.

Best, Christoph

[guenterh]

Thanks for this background information Christoph! With this first I will give it a try again in the way you described it and secondly I'm going to think about it (if it still doesn't solve our problem)

[dr0i]

Hello @guenterh , what is the state of this issue - could you solve it following the hints from @cboehme ? Can we close this issue?

[dr0i]

Closing.