Import command failed to complete

[andrewgilliland]

Brief bug description

When running the @kontent-ai/data-ops import command the script fails to complete. The error Failed to import entity collections. {} appears in the console and the import stops.

Repro steps

1. Run the export command on an Kontent environment. The script runs and returns a .zip file
2. Run the clean command on a different Kontent environment. The environment's data is deleted.
3. Run the import command referencing the .zip from the export, and environment that is empty.

Expected behavior

All content data will be imported into the incoming environment.

Test environment

• Platform/OS: macOS 14.0

Additional context

Logs after running the import script:

Importing: collections
Failed to import entity collections.  {}
Stopping import...

[IvanKiral]

Hey @andrewgilliland.

Sorry for your inconvenience. Unfortunately, I failed to reproduce the error :/ Could you provide us with more information? Can you check whether the collections are cleaned after the clean command? What is also interesting is that the error is not printed in {}. If it is possible we would be grateful to have .zip you were trying to import.

[IvanKiral]

We have hopefully released a fix for the printing errors. Now it shouldn't print {}.

Can you please try it with version 1.1.1 and see whether the error is printed? :)

[andrewgilliland]

@IvanKiral thank you! Printing the error helped resolve the issue with importing collections. The environment has been cleaned out after the clean command.

I am running into an issue now where the import command failed to import entity languages. The error message says a language with an external ID already exists. I did notice with the clean command does not remove the languages from the environment.

To circumvent this I included the --exclude languages flag. The import got further but errored at the import assets set with the error message Failed to find new id for language by old id \"00000000-0000-0000-0000-000000000000\"

Is there a way to delete languages from an environment?

[IvanKiral]

I am glad that we moved forward @andrewgilliland! :) I am sorry that there is a problem with external_id for languages. Unfortunately, we would like to remove the languages with the clean, however, it is not possible right now (deactivating with the postfix is a workaround) :/

Also, exclude and include right now needs to be used with caution, as some entities are dependant on others (like language variants on languages). That's the reason for Failed to find new id for language by old id \"00000000-0000-0000-0000-000000000000\"

However, we would like to try to fix the problem with the languages external_id. I came up with the solution that if in the "clean" environment language with external_id exists, we can check it out and then just activate this language and change its values to the ones that are in the exported file. The implication of this is that a given language would have an old id (of the language that is currently in the environment). What do you think about this solution?

[andrewgilliland]

Sounds good! I was able to update the is_active, name, and codename properties on the "clean" environment language. But am unable to update the id and external_id. Would that potentially cause reference issues down the road?

This allowed me to get further into the import process, but did error out while importing contentTypes. The validation error received was: "The referenced allowed element '{external id: non-existent-element}' was not found in the content type. Provide a reference to an existing element.

In the data being sent for the contentType that errors, I see the {external_id: non-existent-element}. Are these external_ids generated during the import process?

[IvanKiral]

1. ID and external_ids can't be updated. Well if you use a "clean" environment it shouldn't be a problem
2. Yes, we are referencing the non-existing entities with externa_ids as we were living in a dream world where you could reference non-existent using external_ids and the API would not send the error. This is the case for items and assets tho. However, if I may ask, how did you manage to get this error? :D Elements should be imported first, so it should be there. Did you update the exported file? :)

[andrewgilliland]

After I updated the languages with the management api, I ran the import command. These are the logs after running the command:

Importing: collections
Importing: languages
Importing: taxonomies
Retry attempt '1' from a maximum of '5' retries. Request url: 'https://manage.kontent.ai/v2/projects/xxxx-xxxx-xxxx/taxonomies'
Importing: assetFolders
Retry attempt '1' from a maximum of '5' retries. Request url: 'https://manage.kontent.ai/v2/projects/xxxx-xxxx-xxxx/folders'
Importing: assets
Importing: contentTypeSnippets
Importing: contentTypes

And then it errors. I did not update the export file. Should there be a contentElements.json or elements.json in the exported file?

[IvanKiral]

There might be another scenario, where you had the project in which you had some element presented in allowed elements in custom element, and then you removed that element :) it seems this still keeps the id inside custom element's allowed_elements even though the element was deleted. Is it possible this is the scenario that happened? If not would it be possible to obtain the .zip file. so we could investigate it better?

[andrewgilliland]

There is a possibility that an allowed element in a custom element was deleted. Would there be a way to verify this?

This zip is too big to upload here. How else could I get it to you?

[IvanKiral]

Yes, it is possible to check it manually using MAPI :) if you have a source project, just use retrieve a content type in documentation - you can use the form prepared here. Just fill the envId, mapi key, and reference to the type https://kontent.ai/learn/docs/apis/openapi/management-api-v2/#operation/add-a-content-type. Then you can search the custom element and see what is inside allowed_elements.

EDIT: updated script other option is to use some simple script like this

import { createManagementClient } from "@kontent-ai/management-sdk";

const client = createManagementClient({
  environmentId: '<env-id>',
  apiKey: '<management-api-key>',
});

const types = await client
  .listContentTypes()
  .toAllPromise()
  .then(res => res.data.items);

const snippets = await client
  .listContentTypeSnippets()
  .toAllPromise()
  .then(res => res.data.items);

types.forEach(type => type.elements.forEach(element => {
  if (element.type !== 'custom') {
    return;
  }

  element.allowed_elements.forEach(element => {
    if (!type.elements.includes(element.id)) {
      console.log(`found non-existing element in type ${type.codename} with id ${element.id}`)
    }
  })
}))

snippets.forEach(snippet => snippet.elements.forEach(element => {
  if (element.type !== 'custom') {
    return;
  }

  element.allowed_elements.forEach(element => {
    if (!snippet.elements.includes(element.id)) {
      console.log(`found non-existing element in snippet ${snippet.codename} with id ${element.id}`)
    }
  })
}))

If this will not solve the issue we can transfer the .zip maybe we can use something like google drive, one drive or probably this https://wetransfer.com/ if it is up to 2GB :)

[andrewgilliland]

Thanks for that script!

I did find allowed elements inside custom elements that did not exist. Will I need to use the management api to remove them?

[IvanKiral]

Well if you need a solution right now, you can do it :)

But I think we can filter those elements when importing given type/snippet.

I will try to fix all of the issues we are discussing here with this PR https://github.com/kontent-ai/data-ops/pull/39 :)

[IvanKiral]

I have released version 1.1.2, can you please try it again? :)

[andrewgilliland]

Thank you! I am getting further into the import of the environment, but it does fail at importing contentTypes still.

The validation message states that the content type has an element that references a non-existing asset. I did check the environment I imported into, and the asset does exist as it did with the exported environment, but the id has changed.

With the export and import process, shouldn't the id's be preserved from environment to environment?

[IvanKiral]

No, you can't preserve IDs using MAPI :/

That's weird. Are you still trying to import into a clean environment? Do you use include or exclude? If so, that might be the problem as when importing some entities can be dependent on others. If you are using an empty environment and trying to import all, can you please send me the whole error? :)

Sorry for the inconveniences, as it seems there can be a lot of edge cases :/

[andrewgilliland]

I am importing into a clean environment. The export excludes roles, but the import does not include or exclude anything.

Here's the error message: Failed to import entity contentTypes. {"validationErrors":[{"message":"Element '25b6005f-342e-4915-84e8-194e46b22b6d' contains an image referencing a non-existent asset 8510301c-04ae-4f6a-a459-d9c46362c17f."}],"message":"Bad Request – Content type validation failed. See validation_errors for more information.","requestId":"7ba5e13bea868356","errorCode":2040015,"originalError":{"message":"Request failed with status code 400"}}

From examining the data in the request, that asset id is being referenced in the html of a guidelines element of a content type. There is a data-asset-id attribute on a figure and img tag assigned that id.

Here is that element: {"guidelines":"<p>Use a Content Row to create two-column layouts that can be stacked on top of each other.</p>\\n<figure data-asset-id=\\"8510301c-04ae-4f6a-a459-d9c46362c17f\\"><img src=\\"#\\" data-asset-id=\\"8510301c-04ae-4f6a-a459-d9c46362c17f\\"></figure>","type":"guidelines","id":"7d415b3c-e18f-4317-becd-89f88c7070cd","codename":"guidelines","external_id":"content_row_guidelines"}

Thank you, I appreciate your assistance with this.

[IvanKiral]

Hmm, this one is gonna be a little bit harder :D We are already handling non-existing assets, so there must be another edge case.

However, how did you get that element? did you copy it from the contentTypes.json? We are using regex to get this data-asset-id and it seems you have double escaping data-asset-id=\\"8510301c-04ae-4f6a-a459-d9c46362c17f\\"

it doesn't seem to be present in my file

Didn't you or some formatter of your IDE accidentally formatted it?

[andrewgilliland]

That element was directly from the error, it was in the data being sent in the post request in management api for content types. I did not modify it from the export to import commands.

[IvanKiral]

Oh okay, thank you.

I was able to replicate your error, however I needed to use --exclude assets in the command. Just to be sure, did you also use that in your import run?

I made PR #41, however, I am not gonna release it yet, as I am not sure if the problem still occurs. You can still try to build it yourself and try to do import on the branch https://github.com/kontent-ai/data-ops/tree/fix_deleted_assets_in_guidelines

To use it you need to:

1. clone the repo
2. switch to the given branch git switch origin/fix_deleted_assets_in_guidelines
3. use the command by

   npm run start -- import -e <envId> -k<apiKey> -f filename.zip

If you could please try it out, I would be grateful for your feedback.

[andrewgilliland]

I did not add the --exclude assets flag with the import command.

Thank you! I was able to import most everything into the clean environment with that script, but had this error occur:

Failed to import entity languageVariants. {"validationErrors":[],"message":"Bad Request – Publishing item failed because some of its elements are incomplete.","requestId":"992d414984b9f335","errorCode":4040027,"originalError":{"message":"Request failed with status code 400"}}

In the error, the url it was trying a post request to the management api's publish language variant endpoint (I obscured the project-id): https://manage.kontent.ai/v2/projects/<project-id>/items/ee46630b-2f2b-5c72-a154-c95ec180764c/variants/00000000-0000-0000-0000-000000000000/publish

And the data being sent was undefined.

I looked at the content items and saw that many of the items had workflow status of Not translated.

[JiriLojda]

Hi @andrewgilliland,

Thank you for the patience with resolving the issues. I merged and released @IvanKiral's fix so you can now use it with the latest npm release.

Regarding your latest error with language variants, the problem is not with the body of the request, but with the language variant you exported. It appears that the variant (that the tool tried to publish) contains elements that do not satisfy all the restrictions of the content type on which they are based. This can happen, for example, when you add restrictions into the content type after variants of the type are already published and contain elements that do not satisfy the new restrictions. (e.g. making an element required)

I prepared a change on a branch that would make the tool skip such errors. That means the tool would not publish variants that contain incomplete elements. We will consider in what shape to publish the change, but you can try it right now on this branch https://github.com/kontent-ai/data-ops/tree/skip_incomplete_element_errors_on_publish following the same instructions Ivan gave you for a different branch. Can you please try it from the branch to check whether the import would work for you with the change?

[IvanKiral]

As this issue seems to be inactive, I am closing the issue. Thanks @andrewgilliland for your assistance with fixing some issues.