BUG: migrateLegacyData fails while migrating from 8.3.4

[grizooo]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

Exporting node data...
  Exported 25540 events

Importing assets...
  Imported 0 Assets and 1 Image Variant. Errors: 0

Importing events...
  Imported 25540 events into stream "ContentStream:97f1433f-c1d2-4385-8141-f857bf144592"

Replaying projections
Failed to update and commit highest applied sequence number for subscriber "Neos\ContentGraph\DoctrineDbalAdapter\DoctrineDbalContentGraphProjection".
Please run
Neos\EventStore\DoctrineAdapter\DoctrineCheckpointStorage::setup()

  Type: Neos\EventStore\Exception\CheckpointException
  Code: 1652279375
  File: Packages/Libraries/neos/eventstore-doctrineadapter/src/DoctrineCheckpointSt
        orage.php
  Line: 82

Nested exception:
Transaction commit failed because the transaction has been marked for rollback only.

  Type: Doctrine\DBAL\ConnectionException
  File: Packages/Libraries/doctrine/dbal/lib/Doctrine/DBAL/ConnectionException.php
  Line: 15

Nested exception:
Event Neos\ContentRepository\Core\Feature\NodeVariation\Event\NodePeerVariantWasCreated could not be applied: Target parent node not found.

  Type: Neos\ContentGraph\DoctrineDbalAdapter\Domain\Projection\EventCouldNotBeAppl
        iedToContentGraph
  Code: 1645315274
  File: Packages/Neos/Neos.ContentGraph.DoctrineDbalAdapter/src/Domain/Projection/E
        ventCouldNotBeAppliedToContentGraph.php
  Line: 41

Expected Behavior

Migration done whithout errors.

Steps To Reproduce

Neos:
  Flow:
    i18n:
      defaultLocale: en
  ContentRepository:
    contentDimensions:
      language: # name of the dimension, can be anything
        label: Languages # display in UI
        icon: icon-language # icon for UI

        # The default dimension that is applied when creating nodes without specifying a dimension
        default: en_US

        # The default preset to use if no URI segment was given when resolving languages in the router
        defaultPreset: en_US
        presets:
          en_US:
            label: English
            values:
              - en_US
            uriSegment: ''
          de:
            label: Deutsch
            values:
              - de
            uriSegment: 'de'
          es:
            label: Español
            values:
              - es
            uriSegment: 'es'
          fr:
            label: Français
            values:
              - fr
            uriSegment: 'fr'
          it:
            label: Italiano
            values:
              - it
            uriSegment: 'it'
          br:
            label: Português
            values:
              - br
            uriSegment: 'br'
          ru:
            label: Pусский
            values:
              - ru
            uriSegment: 'ru'
  Neos:
    userInterface:
      defaultLanguage: en
      translation:
        autoInclude:
          Paessler.PaesslerCom:
            - Main
            - 'NodeTypes/*'

Environment

- Flow: 8.3.3
- Neos: 8.3.4
- PHP: 8.2

Anything else?

We tried to remove nodes without parents first -> no success. We started node migration to set node without language to default language (suggested from bernhard) -> no success

up:
  comments: 'Migrate from no node dimensions to default dimension values.'
  migration:
    -
      filters:
        -
          type: 'DimensionValues'
          settings:
            dimensionValues: []
      transformations:
        -
          type: 'SetDimensions'
          settings:
            dimensionValues: []
            addDefaultDimensionValues: true

down:
  comments: 'Migrate nodes with default dimensions to no migrations.'
  migration:
    -
      filters:
        -
          type: 'DimensionValues'
          settings:
            dimensionValues: []
            filterForDefaultDimensionValues: true
      transformations:
        -
          type: 'SetDimensions'
          settings:
            dimensionValues: []
            addDefaultDimensionValues: false

[grizooo]

Target Environment:

neos/neos-development-collection: 9.0.x-dev

[pKallert]

Apparently there is a problem with nodes with the same identifier having different parent nodes.

The problem point is here: https://github.com/neos/neos-development-collection/blob/9.0/Neos.ContentGraph.DoctrineDbalAdapter/src/Domain/Projection/Feature/NodeVariation.php#L320

In detail we can have a look at one of the problem nodes, with the id '96a48c08-0286-47de-bfe2-7607d5cc59a0'.

Here are all the DB entries with the corresponding node identifier:

The german entry has a different parentpath which corresponds to the following identifier. Only one node with this identifier exists:

All of the other nodes have the "same" parent node with the same identifier:

For now, I wrote a migration that changes the identifier of all nodes that have sibilings with a different path, but as the migration is still not running through (for different reasons) I cannot check if this breaks anything.

I am unsure if this is an issue unique to our neos installation or if this is a usual.

My really quick fix quick and dirty migration solution (just for reference, it is not tested and should not be used):

    public function isTransformable(NodeData $node)
    {
        $sibilings = $this->nodeDataRepository->findByNodeIdentifier($node->getIdentifier())->toArray();
        if(empty($sibilings)){
            return false;
        }
        $pathlist = [];
        foreach ($sibilings as $sib){
            if(isset($pathlist[$sib->getPath()])){
                $pathlist[$sib->getPath()]++;
            } else {
                $pathlist[$sib->getPath()] = 1;
            }
        }
        if(count($pathlist) == 1){
            return false;
        }
        if($pathlist[$node->getPath()] == 1){
            var_dump('Siblings: '. count($sibilings) . ' Pathlist: '.json_encode($pathlist));
            return true;
        }
        return false;
    }
    public function execute(NodeData $node)
    {
        $identifier = Algorithms::generateUUID();
        $node->setIdentifier($identifier);
    }

[pKallert]

I could reproduce the bug on Neos Demo with Neos 8.3 and an upgrade to Neos 9.

Steps to reproduce:

1. Create a new Page with name "Testing 1" in english dimension
2. Create content of type "Four column content"
3. Create content of type "Text" inside the columns:
4. Switch to german and chose "create and copy" to create a new page in german
5. Create new Page with name "Testing 2"
6. Inside "Testing 2" create content of type "Three column content"
7. Go to Page "Testing 1" and use the Neos UI "cut" tool to cut & paste the text element to "Testing 2" -> "Three column content" with the following result:
   
8. Update environment to Neos 9
9. Do Database migration with "cr:setup" and "cr:migratelegacydata"

Expected result:

Migration runs through

Actual result:

Migration throws error

[pKallert]

What I found out so far:

The migration creates two events that are interesting in this case, first the NodePeerVariantWasCreated event for german, and then if the parent node is different, it creates a NodeAggregateWasMoved event to move the node variant to a new location:

In this case, the following steps happen

1. Event 233 creates Parent Node '6d5c3f7f-2d45-6c94-b9a3-e0dc19fe93a4' in dimension 'DE'
2. Event 236 creates Child Node 'c414b012-5a3b-4212-aee9-1b283ee61932' in dimension 'DE'
3. Event 263 creates Child Node Variant in dimension en_US
4. Event 265 moves the Child Node Variant in dimension en_US to new location (SucceedingSiblingNodeMoveDestination)

As we can see, when the Event 263 is running, the parent node only exists in one dimension (DE) and only has to exist in one dimension since we move it later on. When replaying the projections, in the 'whenNodePeerVariantWasCreated' method we expect the parent to exists all dimensions the child exists in(which is quite reasonable). The method tries to load the parent for the peerNode to create the hierarchy with the parent.

Solutions that I can think of right now:

• In the migration, if the parent is different, check in the visited nodes if the parent exists in the right dimension. If the parent does not exist in the right dimension, create a new event 'NodeAggregateWithNodeWasCreated' instead of 'NodePeerVariantWasCreated' with new unique identifier. Unsure about this since we create a new node element from 'nothing' here.

• Rethink the whole logic of creating a peerVariant and nodeAggregateWasMoved logic for elements with different parents and instead immediately create a new NodeAggregateWithNodeWasCreated. Disadvantage is that we would create lots of new nodes, but perhaps that is something we want since we cannot be sure that any reference between the nodes needs to exist?

• In the migration, if the parent is different and does not exist in the right dimension, create a 'helper' peerNode in the dimension. Then create a new event for deleting it after the nodeAggregateWasMoved. This kind of feels like a bad workaround but is a thing we could do.

• In the whenNodePeerVariantWasCreated projection, accept that the parent in the dimension can be nonexistent. I cannot think of a use case, but perhaps there is one other than the migration?

Since I am quite new to the whole events thing I would be happy for any input here 😄

[bwaidelich]

@pKallert Thanks so much for the excellent bug report including a PR to reproduce this and sorry for the delay on this one.

First little finding: This is the error the projection runs into (note to self: we urgently need to improve error handling, this was hidden underneath some DBAL transaction exceptions).

I'm still trying to wrap my head around the desired behavior currently

[bwaidelich]

I looked into this issue again together with @kitsunet and unfortunately have to surrender for now because I just can't make the required time and brain power available at the moment..

At least I finally understood the issue (which is not that complicated by itself), to put it into my own words:

• In Neos < 9 it's possible to create a variant of a node in a new location
• With Neos 9, variants always share the same path initially but can be moved individually
• So, to simulate the previous behavior we move the variant after its creation
• But that fails in the described scenario because the (peer) variant can't even be created (because there is no parent node covering the variants' dimension space point)

At first I tried to detect that case and to make the variant into a separate node aggregate underneath the new parent (see 3a01416019379cb31c368b274fd49e30ea87fdf5) but that solution seems somehow wrong because we have to invent new node ids and because the relation to the original aggregate is lost.

Christian and me came to the conclusion that we would probably need to extend the *VariantWasCreated events by some optional newParentNodeAggregateId property, such that the can be created and moved at the same time. But I did not manage to adjust the projection logic accordingly.. Maybe @skurfuerst or @nezaniel can help with that?

[nezaniel]

This should be the content graph before the Cut & Paste operation:

So if I understand the UI's cut&paste behavior correctly, what happens next is that "Text" is moved in from Testing1...col1 to Testing2...col1, but only in German(yellow), which is a valid operation in both Neos 8 and 9.

In the migration, we should do the exact same thing: 1) PeerNodeVariantWasCreated(node: Text, source: en_US, target: de) 2) NodeAggregateWasMoved(node: Text, newParent: col1(Testing2), affected DSPs: [de])

Or in general: If a node aggregate is scattered (meaning it has different parents in different DSPs), we must first completely constitute it with variation events before we can move it partially. It is possible that the source parent does not exist in the variant DSP, meaning it has been deleted there in the meantime. In this case have to do the same: Create the parent via variation, apply the move and other operations, and in the end remove it again. This also resembles closest what happened in the Neos8 CR before