search

OpenHFT / Chronicle-Map

Replicate your Key Value Store across your network, with consistency, persistance and performance.
http://chronicle.software/products/chronicle-map/
Apache License 2.0
2.73k stars 467 forks source link

ChronicleMapBuilder fails when pointed to a k8s emptyDir ephemeral volume #530

Closed WeatherGod closed 3 months ago

WeatherGod commented 3 months ago

Currently trying out an update to some third-party server software, and it is failing to create a ChronicleMap in a cache location that I set up as an emptyDir volume. It is trying to create an 8Mi file and the ephemeral storage limit is 8Gi. However, the following code:

    ChronicleMapBuilder<String, DatasetExt> builder = ChronicleMapBuilder.of(String.class, DatasetExt.class)
        .averageValueSize(averageValueSize).entries(maxDatasets).averageKeySize(averagePathLength)
        .valueMarshaller(DatasetExtBytesMarshaller.INSTANCE).skipCloseOnExitHook(true);

Fails with the following traceback:

java.io.IOException: Not enough space to fallocate 8 MiB to / unallocated was 0 MiB
    at net.openhft.chronicle.hash.impl.VanillaChronicleHash.fallocate(VanillaChronicleHash.java:1123) ~[chronicle-map-3.23.5.jar:3.23.5]
    at net.openhft.chronicle.hash.impl.VanillaChronicleHash.map(VanillaChronicleHash.java:1103) ~[chronicle-map-3.23.5.jar:3.23.5]
    at net.openhft.chronicle.hash.impl.VanillaChronicleHash.createMappedStoreAndSegments(VanillaChronicleHash.java:515) ~[chronicle-map-3.23.5.jar:3.23.5]
    at net.openhft.chronicle.map.ChronicleMapBuilder.createWithNewFile(ChronicleMapBuilder.java:1843) ~[chronicle-map-3.23.5.jar:3.23.5]
    at net.openhft.chronicle.map.ChronicleMapBuilder.createWithFile(ChronicleMapBuilder.java:1741) ~[chronicle-map-3.23.5.jar:3.23.5]
    at net.openhft.chronicle.map.ChronicleMapBuilder.createPersistedTo(ChronicleMapBuilder.java:1586) ~[chronicle-map-3.23.5.jar:3.23.5]
    at thredds.server.catalog.tracker.DatasetTrackerChronicle.open(DatasetTrackerChronicle.java:138) ~[classes/:5.5-SNAPSHOT]
    at thredds.server.catalog.tracker.DatasetTrackerChronicle.<init>(DatasetTrackerChronicle.java:79) [classes/:5.5-SNAPSHOT]
    at thredds.server.catalog.tracker.DatasetTrackerChronicle.<init>(DatasetTrackerChronicle.java:68) [classes/:5.5-SNAPSHOT]
    at thredds.core.ConfigCatalogInitialization.reread(ConfigCatalogInitialization.java:149) [classes/:5.5-SNAPSHOT]
    at thredds.core.ConfigCatalogInitialization.init(ConfigCatalogInitialization.java:131) [classes/:5.5-SNAPSHOT]
    at thredds.server.config.TdsInit.onApplicationEvent(TdsInit.java:149) [classes/:5.5-SNAPSHOT]
    at thredds.server.config.TdsInit.onApplicationEvent(TdsInit.java:67) [classes/:5.5-SNAPSHOT]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178) [spring-context-5.3.32.jar:5.3.32]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171) [spring-context-5.3.32.jar:5.3.32]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) [spring-context-5.3.32.jar:5.3.32]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:429) [spring-context-5.3.32.jar:5.3.32]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:386) [spring-context-5.3.32.jar:5.3.32]
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:949) [spring-context-5.3.32.jar:5.3.32]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:594) [spring-context-5.3.32.jar:5.3.32]
    at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:399) [spring-web-5.3.32.jar:5.3.32]
    at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:278) [spring-web-5.3.32.jar:5.3.32]
    at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) [spring-web-5.3.32.jar:5.3.32]
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4441) [catalina.jar:8.5.100]
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4899) [catalina.jar:8.5.100]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171) [catalina.jar:8.5.100]
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:710) [catalina.jar:8.5.100]
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:687) [catalina.jar:8.5.100]
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:660) [catalina.jar:8.5.100]
    at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1176) [catalina.jar:8.5.100]
    at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1889) [catalina.jar:8.5.100]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:829) [?:?]

xref: https://github.com/Unidata/tds/issues/484

Possibly related. I'm also seeing some warning messages about illegal reflective access operations:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access using Lookup on net.openhft.chronicle.core.Jvm (file:/usr/local/tomcat/webapps/thredds/WEB-INF/lib/chronicle-core-2.23.36.jar) to class java.lang.reflect.
AccessibleObject
WeatherGod commented 3 months ago

Re-reading this, I realized I wasn't clear. The exeption isn't thrown by the builder constructor, it is thrown by the next line:

datasetMap = builder.createPersistedTo(dbFile);

Which dbFile is a string of "/usr/local/tomcat/content/thredds/cache/catalog/chronicle.datasets.dat.1". The emptyDir volume is "/usr/local/tomcat/content/thredds/cache". And I am able to manually create files in this directory larger than 8Mi. I'll also note that the file does get created:

root@...:/usr/local/tomcat/content/thredds/cache# touch catalog/foobar.txt
root@...:/usr/local/tomcat/content/thredds/cache# fallocate -z -l 10M catalog/foobar.txt
root@...:/usr/local/tomcat/content/thredds/cache# ls -lh catalog/
total 11M
-rw-r----- 1 tomcat tomcat   28 Mar 27 20:13 catTracker.dat
-rw-r----- 1 tomcat tomcat 8.6M Mar 27 20:13 chronicle.datasets.dat.1
-rw-r----- 1 tomcat tomcat    0 Mar 27 20:13 datarootTracker.dat
-rw-r--r-- 1 root   root    10M Mar 28 14:54 foobar.txt
JerryShea commented 3 months ago

@WeatherGod which version of Chronicle Map is in use?

We are busy with customer work and can't triage/investigate this in the short term. Any investigation you can do yourself would be very helpful, or if you would like us to prioritise it please review the support offerings at https://chronicle.software/openhft-support/

WeatherGod commented 3 months ago

Looks like it is v3.23.5 of chronicle-map. I'm going to try diving into the traceback and see exactly where in chronicle-map it is failing and see if that gives any indication of what's wrong. My suspicion is that it is mostly working as expected, but it is just getting tripped up on some solvable detail, given that the file does get created and that manual execution of fallocate from the command-line works just fine.

On Thu, Mar 28, 2024 at 8:55 PM Jerry Shea @.***> wrote:

@WeatherGod https://github.com/WeatherGod which version of Chronicle Map is in use?

We are busy with customer work and can't triage/investigate this in the short term. Any investigation you can do yourself would be very helpful, or if you would like us to prioritise it please review the support offerings at https://chronicle.software/openhft-support/

— Reply to this email directly, view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/530#issuecomment-2026408108, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACHF6GJFK2WQUKRLIFLUSLY2S3ZXAVCNFSM6AAAAABFNR5RCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRWGQYDQMJQHA . You are receiving this because you were mentioned.Message ID: @.***>

WeatherGod commented 3 months ago

Been trying to build the checkout of 3.23.5 by doing mvn compile, but it is failing. I don't see any documentation on builds. I get this error:

[ERROR] Failed to execute goal on project chronicle-map: Could not resolve dependencies for project 
net.openhft:chronicle-map:bundle:3.23.5: Failed to collect dependencies at net.openhft:affinity:jar:3.23ea1:
 Failed to read artifact descriptor for net.openhft:affinity:jar:3.23ea1: Could not transfer artifact 
net.openhft:third-party-bom:pom:3.22.4-SNAPSHOT from/to chronicle-enterprise-snapshots
 (https://nexus.chronicle.software/content/repositories/snapshots): Authentication failed for
 https://nexus.chronicle.software/content/repositories/snapshots/net/openhft/third-party-bom/3.22.4-SNAPSHOT/third-party-bom-3.22.4-SNAPSHOT.pom
 401 Unauthorized

I just want to compile a small change I made to src/main/java/net/openhft/chronicle/hash/impl/VanillaChronicleHash.java into a jar that I can place in the WEB-INF/ to help me diagnose what's going on. Thank you.

WeatherGod commented 3 months ago

I have confirmed that #519 fixes this problem. I will advise the maintainers of the other project to update their dependency.

WeatherGod commented 3 months ago

Is there a rough estimate for the next non-Early Access release (days, weeks, months)? This will help the developers of the other project decide whether or not to hold off on putting out their next release. They want to only use non-Early Access releases for their dependencies.