unable to add mongodb data source

[seem8]

Issue Summary

I am unable to add MongoDB data source.

Steps to Reproduce

1. Add data source.
2. Choose MongoDB.
3. Fill mandatory fields with valid MongoDB url, or just 'aaa'.

Any other info e.g. Why do you consider this to be a bug? What did you expect to happen instead? I would like to add MonoDB data source. It works great with Clickhouse and MySQL sources.

Technical details:

• Redash Version: 9.0.0
• Browser/OS: Firefox/Ubuntu 18.04
• How did you install Redash: https://redash.io/help/open-source/dev-guide/setup

Traceback (from virtualenv with requirements and requrements_all installed): python3 manage.py ds new --type mongodb Name: aaa Connection String (required): aaa Database Name (required): aaa Replica Set Name (optional): Replica Set Read Preference (optional):
Traceback (most recent call last): File "manage.py", line 9, in manager() File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 722, in call return self.main(*args, kwargs) File "/data/redash/app/lib/python3.6/site-packages/flask/cli.py", line 586, in main return super(FlaskGroup, self).main(args, kwargs) File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(args, kwargs) File "/data/redash/app/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, kwargs) File "/data/redash/app/lib/python3.6/site-packages/flask/cli.py", line 426, in decorator return __ctx.invoke(f, *args, *kwargs) File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(args, kwargs) File "/data/redash/app/gitredash/redash/cli/data_sources.py", line 155, in new if not options.is_valid(): File "/data/redash/app/gitredash/redash/utils/configuration.py", line 45, in is_valid self.validate() File "/data/redash/app/gitredash/redash/utils/configuration.py", line 57, in validate jsonschema.validate(self._config, self._schema) File "/data/redash/app/lib/python3.6/site-packages/jsonschema/validators.py", line 936, in validate cls.check_schema(schema) File "/data/redash/app/lib/python3.6/site-packages/jsonschema/validators.py", line 300, in check_schema raise exceptions.SchemaError.create_from(error) jsonschema.exceptions.SchemaError: <map object at 0x7f9464015ac8> is not of type 'array'

Failed validating 'type' in metaschema['properties']['properties']['additionalProperties']['properties']['enum']: {'items': True, 'type': 'array'}

On schema['properties']['readPreference']['enum']: <map object at 0x7f9464015ac8>

I modified 2 files to get more debug data: /data/redash/app/lib/python3.6/site-packages/jsonschema/validators.py: @classmethod def check_schema(cls, schema): for error in cls(cls.META_SCHEMA).iter_errors(schema):

ADDED DEBUG

            with open('/tmp/sprjson','w+') as sprjson:
                sprjson.write(str(schema))
                sprjson.write(str(cls))
                sprjson.write(str(error))
            # / ADDED DEBUG
            raise exceptions.SchemaError.create_from(error)

/data/redash/app/gitredash/redash/utils/configuration.py def validate(self):

ADDED DEBUG

    with open('/tmp/validebug','w+') as validebug:
        validebug.write(str(self._config))
        validebug.write(str(self._schema))
        # / ADDED DEBUG
    jsonschema.validate(self._config, self._schema)

cat /tmp/sprjson {'type': 'object', 'properties': {'connectionString': {'type': 'string', 'title': 'Connection String'}, 'dbName': {'type': 'string', 'title': 'Database Name'}, 'replicaSetName': {'type': 'string', 'title': 'Replica Set Name'}, 'readPreference': {'type': 'string', 'title': 'Replica Set Read Preference', 'enum': <map object at 0x7f9464015ac8>}}, 'required': ['connectionString', 'dbName']}<class 'jsonschema.validators.create..Validator'><map object at 0x7f9464015ac8> is not of type 'array'

Failed validating 'type' in schema['properties']['properties']['additionalProperties']['properties']['enum']: {'items': True, 'type': 'array'}

On instance['properties']['readPreference']['enum']: <map object at 0x7f9464015ac8>

cat /tmp/validebug {'connectionString': 'aaa', 'dbName': 'aaa'}{'type': 'object', 'properties': {'connectionString': {'type': 'string', 'title': 'Connection String'}, 'dbName': {'type': 'string', 'title': 'Database Name'}, 'replicaSetName': {'type': 'string', 'title': 'Replica Set Name'}, 'readPreference': {'type': 'string', 'title': 'Replica Set Read Preference', 'enum': <map object at 0x7f9464015ac8>}}, 'required': ['connectionString', 'dbName']}

[seem8]

I've found it. It's a really monkey patch, but it worked. Line 163 of a file /data/redash/app/gitredash/redash/handlers/data_sources.py:

schema = get_configuration_schema_for_query_runner_type(req["type"]) was chaned to changed to schema = { 'type': 'object', 'properties': {'connectionString': {'type': 'string', 'title': 'Connection String'}, 'dbName': {'type': 'string', 'title': 'Database Name'}, 'replicaSetName': {'type': 'string', 'title': 'Replica Set Name'}, 'readPreference': {'type': 'string', 'title': 'Replica Set Read Preference', 'enum': '8'}}, 'required': ['connectionString', 'dbName'], }

I need that 8, because a new data source will have id of 8 in the database.

Next is config variable below. Line: config = ConfigurationContainer(filter_none(req["options"]), schema) was changed to config = { 'connectionString': "mongodb://USER:PASS@IP.IP.IP.IP/?authSource=DATABASE&gssapiServiceName=mongodb", 'dbName': 'DATABASE_NAME', }

I also commented out this two lines:

if not config.is_valid():

    #    abort(400)

I was able to add MongoDB databases.

[wtfiwtz]

Is it missing a query_runner? I had the same issue with version 10.1.0 with Athena. The abort(400) responses with a BadRequest error - The browser (or proxy) sent a request that this server could not understand. which is very generic.

[wtfiwtz]

Each Query Runner is within redash/query_runners and ends with a statement like

register(Athena)

So check your Python dependencies are loaded correctly.

Look for this error message:

        logger.debug(
            "%s query runner enabled but not supported, not registering. Either disable or install missing "
            "dependencies.",
            query_runner_class.name(),
        )

[justinclift]

Hmmm, this is a old bug report for a pretty old release of Redash.

With our latest development code (eg the redash/redash:preview image on Docker Hub), we definitely have MongoDB working as a data source.

@wtfiwtz If you're not running a production setup, it might be worth trying that out? Note that the image isn't well tested enough (yet) to consider "production ready", so be careful with it. :smile:

[wtfiwtz]

Thanks @justinclift ... managed to get it working on 10.1.0 (which we have in production 😬... hopefully can upgrade this shortly)

I still get a warning about pyarrow versions but this is working with requirements_all_ds.txt

google-api-python-client==1.7.11
protobuf==3.17.3
gspread==3.1.0
impyla==0.16.0
influxdb==5.2.3
mysqlclient==1.3.13
oauth2client==4.1.3
pyhive==0.6.1
pymongo[tls,srv]==3.9.0
vertica-python==0.9.5
td-client==1.0.0
pymssql==2.1.4
dynamo3==0.4.10
sasl>=0.1.3
cassandra-driver==3.21.0
atsd_client==3.0.5
simple_salesforce==0.74.3
PyAthena==1.5.0
qds-sdk>=1.9.6
ibm-db>=2.0.9
pydruid==0.5.7
requests_aws_sign==0.1.5
phoenixdb==0.7
# certifi is needed to support MongoDB and SSL:
certifi<2021.0.0
pydgraph==2.0.2
pyexasol==0.12.0
python-rapidjson==0.8.0
pyodbc==4.0.28
trino==0.305
cmem-cmempy==21.2.3
xlrd==2.0.1
openpyxl==3.0.7

firebolt-sqlalchemy==0.0.9
firebolt-sdk==0.2.1
pytz<2021.0
croniter~=1.0
python-dateutil==2.8.0
botocore==1.13
boto3==1.10.0
paramiko==2.7.0
memsql==3.0.0
snowflake-connector-python==2.3.0
azure-kusto-data==0.0.35
dql>=0.5.26

pyarrow<0.14
pymapd>=0.19.0
thrift>=0.11.0
thrift_sasl>=0.1.0

I run docker-compose run --rm server pip install -r requirements_all_ds.txt to check

Thanks!

[justinclift]

Well, as long as it's working I suppose. :grin:

[justinclift]

I'll close this issue report as it's for a very outdated Redash release, and you've managed to fix your setup as well. :smile: