apache / accumulo

Apache Accumulo
https://accumulo.apache.org
Apache License 2.0
1.07k stars 445 forks source link

Add ZooKeeper Namespace Id-to-name mapping #4996

Closed meatballspaghetti closed 5 days ago

meatballspaghetti commented 1 month ago

This PR is part of an ongoing work to improve efficiency and reduce slowdown of retrieving and writing namespace names and table names in ZooKeeper. These particular changes modify the namespace names, and a follow-up PR in the near future will modify the table names. It replaces the use of the /name nodes under namespaces with a namespace id-to-name mapping in a Json format stored under the /namespaces node. With this, when any function wants to retrieve a namespace name, ZooKeeper will only need to read the single mapping instead of iterating through all /name nodes every time.

New files added for implementation:

Files changed for implementation:

Files changed for upgrading:

Original Issue: #4698

meatballspaghetti commented 1 month ago

In progress of fixing Upgrader11to12Test which failed the unit test checks.

meatballspaghetti commented 1 month ago

Upgrader11to12Test has been fixed and passes now.

meatballspaghetti commented 2 weeks ago

Commit dd8c4fa has moved all namespace mapping actions (create/put, rename, remove) into NamespaceMapping, ensured that mutateExisting() is used in all cases, and gives checks for non-existent/already-existing namespaces (depending on the action). The actual namespace node creation and removal functionalities remain in their respective classes and occur after the mapping is updated.

dlmarion commented 4 days ago

@meatballspaghetti - please take a look at NamespacesIT, I'm seeing 4 tests fail with the same type of error:

java.lang.IllegalArgumentException: Multiple entries with same key: =+default and =x
    at com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:382)
    at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:376)
    at com.google.common.collect.ImmutableSortedMap.fromEntries(ImmutableSortedMap.java:560)
    at com.google.common.collect.ImmutableSortedMap.access$100(ImmutableSortedMap.java:68)
    at com.google.common.collect.ImmutableSortedMap$Builder.buildOrThrow(ImmutableSortedMap.java:739)
    at com.google.common.collect.ImmutableSortedMap$Builder.build(ImmutableSortedMap.java:718)
    at org.apache.accumulo.core.clientImpl.NamespaceMapping.update(NamespaceMapping.java:148)
    at org.apache.accumulo.core.clientImpl.NamespaceMapping.getNameToIdMap(NamespaceMapping.java:160)
    at org.apache.accumulo.core.clientImpl.Namespaces.getNamespaceId(Namespaces.java:83)
    at org.apache.accumulo.core.clientImpl.Namespaces.lookupNamespaceId(Namespaces.java:97)
    at org.apache.accumulo.core.clientImpl.Namespaces.namespaceNameExists(Namespaces.java:110)
    at org.apache.accumulo.core.clientImpl.NamespaceOperationsImpl.exists(NamespaceOperationsImpl.java:113)
    at org.apache.accumulo.test.NamespacesIT.createAndDeleteNamespace(NamespacesIT.java:220)
ctubbsii commented 4 days ago

I'm seeing 4 tests fail with the same type of error:

I didn't see that specific stack trace when running NamespacesIT in the 3.1 branch. I got different exceptions instead. Perhaps that stack trace only appears in the main branch? I did fix the test failures I saw in 3.1 with #5067. I tested a local merge to the main branch and NamespacesIT passed there also.