lastpass-vault-item-import.py fails to create items and op give `unable to list item categories` error

[scottisloud]

Problem

1. Script fails to import some items after certain point - likely special characters in extra field. Likely related to #21
2. unable to build a list of item categories error from op during import. Customer experienced that error and was then unable to log into 1Password, perhaps due to rate limiting.

Expected behaviour

Import script should import all items that are not secure notes.

Actual Behaviour

The script fails to import all exported data.

Additional details

Report from customer:

the script lastpass-vault-item-import.py worked for my export (13 items), but ran into issues when trying to import the export.csv from [other] account (460 items, most of which are shared folder lastpass credentials). It seems to have processed some of the export file, but then produced the below message 223 times:

[ERROR] 2023/01/11 01:08:29 Unable to build a list of item categories.

I'm going to venture to guess that it ran into issues with the "extra" column data which has some special characters in some of our shared secrets.

Interestingly while the error message was being thrown at me in the terminal, I was unable to login to the 1password portal and it returned an httpd status code 429 (too many requests). So the python script seems to have hammered on the API and gotten itself blocked/rate limited. Now that the errors have stopped, when I login to the portal and check "all items" I'm seeing 114 items. So it imported something, but not everything. And it didn't create any vaults. :-(

and

I did now see that it created a vault called "0" and tagged the secrets with "0" and there are 100 items in that vault. Is that the expected behavior?

This is probably several, distinct, issues.

[scottisloud]

Was unable to reproduce any issues associated with a non-delimiting comma since merging #23. That alone may not be the case.

The unable to build a list of item categories may result from either a session timing out (e.g., if op is unlocked and idle too long before running the script`, or perhaps due to rate limiting. Further investigation required to see what size of import might lead to rate limiting.

Will continue to explore both issues as time permits.

[scottisloud]

Possibilities:

• 1Password/CLI locked prior to, or while, the script was running. This did not reproduce ⛔

  • When this happens, 1Password for Mac prompts to re-auth several times, which if cancelled, results in:

    [ERROR] 2023/01/16 12:44:47 unable to process line 1: authorization prompt dismissed, please try again
    [ERROR] 2023/01/16 12:44:47 unable to process line 1: You've tried to unlock with the 1Password app too many times. Try again in 15 seconds.

• Hitting a rate limit on 1Password servers: ⚠️

  • I doubt this would be the case. Rate limit for item creation is quite high, and I have run tests involving the creation of tens of thousands of items via the CLI without rate limiting (it also took hours), so although I haven't confirmed it, I don't think that's the cause of the error either.

More investigation needed but I am thinking this may be an environment-specific edge-case, or something else that's really in the margins.

[scottisloud]

At this point I have seen no further reports of this issue and cannot reproduce. Closing.

[scottisloud]

A second report (internal number: 103457) of this has come in. Re-opening but still no repro steps.

[scottisloud]

Closing as this script now serves mostly as an example for what's possible and has been largely rendered redundant by the importer built into 1Password.