Open Adityak4201 opened 1 year ago
justinclift
commented
1 year ago Interesting. Yeah, that pymongo version of 3.9.0 is even in the latest Redash source code:
We're updating other dependencies over the last days and weeks anyway, so we might as well try a newer pymongo and see if it builds without needing other changes.
justinclift
commented
1 year ago @Adityak4201 I've just created a PR with that change, mostly to see if it passes our test suite. No idea if it'll work or not, but it's worth trying out just in case. :smile:
justinclift
commented
1 year ago The newer PyMongo passed the current Redash test suite, though I'm not sure how much that actually tests the Mongo part of things. That being said, I've merged it the PR into our master branch, so it's currently re-running the test suite on there + it'll automatically build a new preview docker image:
https://hub.docker.com/r/redashcommunity/redash/tags
You might need to give it an hour or so to finish the tests and building, then you could try that docker build out with your Mongo version 6 data source and see if it works ok. :smile:
justinclift
commented
1 year ago It looks like the updated docker hub image has already built, so you should be ok to try it out. :smile:
Adityak4201
commented
1 year ago Hey @justinclift. Thanks for the quick response. I will test it and let you know
justinclift
commented
1 year ago Thanks, please do. If the updated PyMongo library doesn't work properly, we'll probably need to change the version back to the older one until we can look at it properly.
Adityak4201
commented
1 year ago I have been trying to run the https://hub.docker.com/r/redashcommunity/redash/tags with docker. I have been following https://redash.io/help/open-source/dev-guide/docker for running the application on docker.
But, when I am running the command docker-compose run --rm server create_db, I am getting the following error:
Starting redash_redis_1 ... done
Starting redash_postgres_1 ... done
Traceback (most recent call last):
File "/app/manage.py", line 6, in
justinclift
commented
1 year ago Interesting. It looks like an error happening in the DynamoDB connection library. Looking at that library on GitHub (https://github.com/stevearc/dql), it seems pretty much dead or close to it.
My first thought is we could just remove the DynamoDB library support for now, in order to get things working.
But it might actually be something else (simple) instead.
@gaecoli Would you have a few minutes to take a look at this?
gaecoli
commented
1 year ago Interesting. It looks like an error happening in the DynamoDB connection library. Looking at that library on GitHub (https://github.com/stevearc/dql), it seems pretty much dead or close to it.
My first thought is we could just remove the DynamoDB library support for now, in order to get things working.
But it might actually be something else (simple) instead.
@gaecoli Would you have a few minutes to take a look at this?
Tonight, i will look at this. I'm busy right now. Sorry.
justinclift
commented
1 year ago Tonight is completely fine, thank you. :smile:
gaecoli
commented
1 year ago I have been trying to run the https://hub.docker.com/r/redashcommunity/redash/tags with docker. I have been following https://redash.io/help/open-source/dev-guide/docker for running the application on docker.
But, when I am running the command
docker-compose run --rm server create_db, I am getting the following error:Starting redash_redis_1 ... done Starting redash_postgres_1 ... done Traceback (most recent call last): File "/app/manage.py", line 6, in from redash.cli import manager File "/app/redash/init.py", line 56, in import_query_runners(settings.QUERY_RUNNERS) File "/app/redash/query_runner/init.py", line 445, in import_query_runners import(runner_import) File "/app/redash/query_runner/dynamodb_sql.py", line 10, in from dql import Engine, FragmentEngine File "/usr/local/lib/python3.7/site-packages/dql/init.py", line 6, in from .cli import DQLClient File "/usr/local/lib/python3.7/site-packages/dql/cli.py", line 21, in from .engine import FragmentEngine File "/usr/local/lib/python3.7/site-packages/dql/engine.py", line 47, in from .grammar import line_parser, parser File "/usr/local/lib/python3.7/site-packages/dql/grammar/init.py", line 37, in from .query import ( File "/usr/local/lib/python3.7/site-packages/dql/grammar/query.py", line 45, in from .parsed_primitives import primitive as parsed_primitive File "/usr/local/lib/python3.7/site-packages/dql/grammar/parsed_primitives.py", line 30, in integer = pyparsing_common.signedInteger AttributeError: type object 'pyparsing_common' has no attribute 'signedInteger'
Please show me how do you connect it.
gaecoli
commented
1 year ago Interesting. It looks like an error happening in the DynamoDB connection library. Looking at that library on GitHub (https://github.com/stevearc/dql), it seems pretty much dead or close to it.
My first thought is we could just remove the DynamoDB library support for now, in order to get things working.
But it might actually be something else (simple) instead.
@gaecoli Would you have a few minutes to take a look at this?
I think our docker image not build dql==0.6.2 in it. Thus causing the situation. @justinclift
justinclift
commented
1 year ago Oh. I wonder if our docker image build is not using the requirements_all_ds.txt during build then...
gaecoli
commented
1 year ago Oh. I wonder if our docker image build is not using the requirements_all_ds.txt during build then...
Yes, you understand me.
justinclift
commented
1 year ago Cool. We need to fix that. :smile:
@junnplus @hazmeister It looks like our preview images on Docker Hub aren't being built with all of the libraries from requirements_all_ds.txt. Any idea how to get them included?
justinclift
commented
1 year ago Hmmm, it looks like GitHub now enforces a very narrow "maximum width" for wiki pages, making them useless for tables. That's not good. :frowning:
justinclift
commented
1 year ago I think there's a cut-n-paste mistake here too:
Both REFERRER_POLICY and FEATURE_POLICY are being loaded from the environment variable REDASH_REFERRER_POLICY, even though they have different uses and take different values. It probably hasn't been spotted before due to the default value for each of them being ok.
That FEATURE_POLICY one should probably be using a REDASH_FEATURE_POLICY environment variable instead.
gaecoli
commented
1 year ago Yes. It will be fixed.
On Jun 20, 2023, at 18:47, Aditya Khandelwal @.***> wrote:
Also, while I have been trying to run docker image for redashcommunity/redash:preview, I am getting error:
2023-06-20 10:43:44.157 UTC [1] FATAL: database files are incompatible with server 2023-06-20 10:43:44.157 UTC [1] DETAIL: The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 15.3.
I think postgresql version needs to be cross checked again.
— Reply to this email directly, view it on GitHub https://github.com/RedashCommunity/redash/issues/59#issuecomment-1598540673, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALBBQNGDJG4OJRDOM3IOOBTXMF5UBANCNFSM6AAAAAAZLXQQD4. You are receiving this because you were mentioned.
justinclift
commented
1 year ago @jezdez Back in the day (~2019) you added a commit for Redash with this code fragment in it:
That code fragment seems to have a cut-n-paste typo in it.
Any idea if FEATURE_POLICY should be getting picked up from REDASH_FEATURE_POLICY rather than from REDASH_REFERRER_POLICY?
Not sure if you remember this code in question, but figured it doesn't hurt to ask... :smile:
jezdez
commented
1 year ago Yeah, definitely a typo, REDASH_FEATURE_POLICY is the correct one
justinclift
commented
1 year ago Awesome, thanks heaps @jezdez. :smile:
It'll be fixed in #64, which is running through the CI tests now.
hazmeister
commented
1 year ago Cool. We need to fix that. :smile:
@junnplus @hazmeister It looks like our preview images on Docker Hub aren't being built with all of the libraries from requirements_all_ds.txt. Any idea how to get them included?
Any ideas what ds stands for? I'm a few days without access to a proper machine- can you pip install it manually using docker exec? I seem to recall python needs updating- perhaps that version only installs on a newer python base?
justinclift
commented
1 year ago DS means "Data Source". :smile:
I can probably take a look at it tomorrow. Badly need sleep at the moment after being up all night again. :man_facepalming:
gaecoli
commented
1 year ago DS means "Data Source". 😄
I can probably take a look at it tomorrow. Badly need sleep at the moment after being up all night again. 🤦♂️
We can move ds lib to requirement.txt. Just a idea.
gaecoli
commented
1 year ago Cool. We need to fix that. 😄 @junnplus @hazmeister It looks like our preview images on Docker Hub aren't being built with all of the libraries from requirements_all_ds.txt. Any idea how to get them included?
Any ideas what ds stands for? I'm a few days without access to a proper machine- can you pip install it manually using docker exec? I seem to recall python needs updating- perhaps that version only installs on a newer python base?
When you build images with docker, i want to know what args are you used?
justinclift
commented
1 year ago In theory, it should be done by this:
https://github.com/RedashCommunity/redash/blob/master/.github/workflows/docker-preview.yml
That file seems to be mostly pre-defined GitHub Action er... "actions". I'm not sure if the docker image build is being done by docker/setup-buildx-action@v2 or docker/build-push-action@v4 though.
Those actions have docs:
But after skimming those, it's still unclear if the build action is just directly running through the Dockerfile to create the image, or perhaps it's using the Makefile.
If it's building the image directly using the Dockerfile, then I thought it would be automatically using requirements_all_ds.txt already:
That's supposed to install all of the data source dependencies unless the variable "skip_ds_deps" is set to any value and passed to it.
When I run make by itself in my local Redash repo, it executes:
COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose buildThat's the compose_build target in the Makefile:
Which will be using the docker-compose.yml file:
Which in turn will build the docker image from the Dockerfile if it's not already created by the GitHub actions run this time around.
To me, I'd think this would also be including all of the data source library dependencies already as per the above. It uses the same Dockerfile for the build.
Alternatively, there's the build process available through yarn instead:
That seems to be for just the front end though, so is probably called by other stuff as part of the build.
We can move ds lib to requirement.txt.
Yeah, it's probably the fastest way to get to a working result, while we're still getting a better idea of how all the pieces fit together.
a) Make it work <-- we're still here b) Optimise it
:wink:
justinclift
commented
1 year ago Initial draft PR created. No idea if it'll pass our CI though. We may need to drop some of the data sources which pip (in my local install) complained about.
Lets see how it goes... :smile:
justinclift
commented
1 year ago @Adityak4201 In the main getredash/redash repo we have a potential update for this almost ready. But I'd like someone that uses MongoDB 6 to try it and make sure the data source is indeed working ok (once the update is merged).
Is that something you'd be ok doing?
Adityak4201
commented
1 year ago @justinclift I have checked the connection to mongoDB 6 from getredash/redash and it was successful. I have successfully built dashboard also.
Thanks for the quick update :smile:
I have mongoDB 6.0. When I am trying to connect it from redash 8.0, I am getting error:
Unsupported OP_QUERY command: connectionStatus. The client driver may require an upgrade. For more details, see https://dochub.mongodb.org/core/legacy-opcode-removal.
After going through redash's code, I realized they are using pymongo 3.9.0 which only supports till mongo 4.2. Please upgrade the pymongo version to 4.3.3