[V3 Config MongoDB] Unable to set api key

[fixator10]

Other bugs

What were you trying to do?

Set token for service via [p]set token or bot.set_shared_api_tokens()

What were you expecting to happen?

Save token to database without exception

What actually happened?

PyMongo raises exception pymongo.errors.OperationFailure

Traceback (most recent call last):
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1385, in _retry_with_session
    return func(session, sock_info, retryable)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/collection.py", line 1161, in _delete
    retryable_write=retryable_write)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/collection.py", line 1147, in _delete
    retryable_write=retryable_write)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/pool.py", line 613, in command
    user_fields=user_fields)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/network.py", line 167, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/helpers.py", line 159, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Transaction numbers are only allowed on a replica set member or mongos
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/drivers/mongo.py", line 226, in set
    await mongo_collection.delete_many(pkey_filter, session=session)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/collection.py", line 1240, in delete_many
    collation=collation, session=session),
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/collection.py", line 1165, in _delete_retryable
    _delete, session)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1492, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1426, in _retry_with_session
    raise OperationFailure(errmsg, exc.code, exc.details)
pymongo.errors.OperationFailure: This MongoDB deployment does not support retryable writes. Please add retryWrites=false to your connection string.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/discord/ext/commands/core.py", line 79, in wrapped
    ret = await coro(*args, **kwargs)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/core_commands.py", line 1149, in api
    await ctx.bot.set_shared_api_tokens(service, **tokens)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/bot.py", line 529, in set_shared_api_tokens
    group.update(tokens)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/config.py", line 88, in __aexit__
    await self.value_obj.set(self.raw_value)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/config.py", line 534, in set
    await super().set(value)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/config.py", line 230, in set
    await self.driver.set(self.identifier_data, value=value)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/drivers/mongo.py", line 275, in set
    (pymongo.InsertOne(d) for d in to_insert if d),
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/drivers/mongo.py", line 275, in <genexpr>
    (pymongo.InsertOne(d) for d in to_insert if d),
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/drivers/mongo.py", line 304, in generate_documents_to_insert
    document["_id"] = {"RED_uuid": uuid, "RED_primary_key": primary_keys + [pkey]}
TypeError: 'str' object does not support item assignment

This also creates collection with name %DBNAME%.Core.SHARED_API_TOKENS (according to other configs, this should be just Core.SHARED_API_TOKENS) Created collection is empty.

How can we reproduce this issue?

1. Setup bot ("develop" version, latest github commit) and use MongoDB as config driver
2. Try to set api token via command or bot's function

[mikeshardmind]

This may be related to #2821 , but I'd like to get a bit more info for looking into this. (especially since I thought we had a fallback strategy for this specific error)

If you have the following info:

• Mongo version:
• Host OS

I'll look into this more.

[fixator10]

MongoDB shell version v4.2.0
MongoDB server version: 4.2.0
db.version(): 4.2.0

lsb_release -a:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

Same happened with 4.0.12 MongoDB

[mikeshardmind]

My apologies for not getting back to you on this sooner, all of the information below was more recently known than my response and I lost track of getting back to you on this.

Root cause: We require transaction or retry writes (the latter as a fallback) from the mongo instance in use. The failure didn't show with core prior to this, as prior to this, there would never be an attempt in core to set a value above the document level in the underling mongo structure.

As for handling this, we should document that requirement as for most users, this will mean use of mongos Note: This does mean doing this should solve the issue in your specific case even prior to the proposed full remedies

There should also likely be detection of the lack of functionality in code prior to bot start-up to prevent mid execution breakage caused by lacking the functionality.

[fixator10]

After converting DB to Replica set, exception traceback changed:

Traceback (most recent call last):
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/dev_commands.py", line 171, in _eval
    result = await func()
  File "<string>", line 3, in func
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/bot.py", line 529, in set_shared_api_tokens
    group.update(tokens)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/config.py", line 88, in __aexit__
    await self.value_obj.set(self.raw_value)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/config.py", line 534, in set
    await super().set(value)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/config.py", line 230, in set
    await self.driver.set(self.identifier_data, value=value)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/drivers/mongo.py", line 231, in set
    session=session,
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/collection.py", line 757, in insert_many
    blk.ops = [doc for doc in gen()]
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/collection.py", line 757, in <listcomp>
    blk.ops = [doc for doc in gen()]
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/pymongo/collection.py", line 747, in gen
    for document in documents:
  File "/home/fixator/Red-V3/lib/python3.7/site-packages/redbot/core/drivers/mongo.py", line 304, in generate_documents_to_insert
    document["_id"] = {"RED_uuid": uuid, "RED_primary_key": primary_keys + [pkey]}
TypeError: 'str' object does not support item assignment

This is still happens on either [p]set api and bot.set_shared_api_tokens

[ghost]

I also have this error. After converting DB to Replica set

[NNTin]

Experiencing same issue:

https://github.com/Cog-Creators/Red-DiscordBot/blob/8267ad9aabc15b9c339ae54a8a7acad9b2917508/redbot/core/drivers/mongo.py#L55 https://docs.mlab.com/faq/#why-am-i-getting-the-transaction-numbers-are-only-allowed-on-storage-engines-that-support-document-level-locking-error

I cannot use mlab as my mongodb server because of this error.

Traceback:

(venv) root@vultr:~/Red-kun# python -m redbot heroku --token $TOKEN --prefix !
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/venv/lib/python3.7/site-packages/redbot/__main__.py", line 179, in <module>
    main()
  File "/root/venv/lib/python3.7/site-packages/redbot/__main__.py", line 115, in main
    loop.run_until_complete(red.maybe_update_config())
  File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete
  File "/root/venv/lib/python3.7/site-packages/redbot/core/bot.py", line 147, in maybe_update_config
    await self.db.schema_version.set(schema_version)
  File "/root/venv/lib/python3.7/site-packages/redbot/core/config.py", line 165, in set
    await self.driver.set(self.identifier_data, value=value)
  File "/root/venv/lib/python3.7/site-packages/redbot/core/drivers/red_mongo.py", line 164, in set
    upsert=True,
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/venv/lib/python3.7/site-packages/pymongo/collection.py", line 1002, in update_one
    session=session),
  File "/root/venv/lib/python3.7/site-packages/pymongo/collection.py", line 858, in _update_retryable
    _update, session)
  File "/root/venv/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1280, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/root/venv/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1233, in _retry_with_session
    return func(session, sock_info, retryable)
  File "/root/venv/lib/python3.7/site-packages/pymongo/collection.py", line 854, in _update
    retryable_write=retryable_write)
  File "/root/venv/lib/python3.7/site-packages/pymongo/collection.py", line 824, in _update
    retryable_write=retryable_write).copy()
  File "/root/venv/lib/python3.7/site-packages/pymongo/pool.py", line 584, in command
    user_fields=user_fields)
  File "/root/venv/lib/python3.7/site-packages/pymongo/network.py", line 158, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/root/venv/lib/python3.7/site-packages/pymongo/helpers.py", line 155, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Transaction numbers are only allowed on storage engines that support document-level locking

[mikeshardmind]

connection string isn't the actual issue here, it's just masking the actual issue for people who don't have retry writes available. Actual issue is shown in Fixator's last comment here, and is related to sets in config which would require writes to multiple documents.

[mikeshardmind]

After more was looked into with this, and a lot of discussion about various ways of handling it, the answer is that we are going to need to drop mongo support. Probably in 3.2

There's more to it than just the API keys, this was just the part which pointed out the issue clearly. More details will be added in #3099 about this. Leaving this open as an informational issue for now.

For those wanting a DB backend for config, postgres is and and will continue to be supported

[mikeshardmind]

3099 was the removal of the mongo driver, closing this.