Open anuj-kumar1SF opened 2 months ago
github-actions[bot]
commented
2 months ago Hey @anuj-kumar1SF! 👋 Thanks for flagging this bug! 🐛🔍
You're our superhero bug hunter! 🦸♂️🦸♀️ Before we suit up to squash this bug, could you please:
📚 Double-check our documentation: https://rengine.wiki 🕵️ Make sure it's not a known issue 📝 Provide all the juicy details about this sneaky bug
Once again - thanks for your vigilance! 🛠️🚀
its-mr-monday
commented
2 months ago During setup either the rengine user is not being created in postgresql or it's password is not set properly
Here's the error output:
Traceback (most recent call last):
File "/usr/src/app/manage.py", line 26, in <module>
main()
File "/usr/src/app/manage.py", line 22, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/core/management/commands/migrate.py", line 75, in handle
self.check(databases=[database])
File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 419, in check
all_issues = checks.run_checks(
File "/usr/local/lib/python3.10/dist-packages/django/core/checks/registry.py", line 76, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
File "/usr/local/lib/python3.10/dist-packages/django/core/checks/model_checks.py", line 34, in check_all_models
errors.extend(model.check(**kwargs))
File "/usr/local/lib/python3.10/dist-packages/django/db/models/base.py", line 1303, in check
*cls._check_indexes(databases),
File "/usr/local/lib/python3.10/dist-packages/django/db/models/base.py", line 1695, in _check_indexes
connection.features.supports_covering_indexes or
File "/usr/local/lib/python3.10/dist-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/postgresql/features.py", line 92, in is_postgresql_11
return self.connection.pg_version >= 110000
File "/usr/local/lib/python3.10/dist-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/postgresql/base.py", line 329, in pg_version
with self.temporary_connection():
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 603, in temporary_connection
with self.cursor() as cursor:
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 259, in cursor
return self._cursor()
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 235, in _cursor
self.ensure_connection()
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 218, in ensure_connection
with self.wrap_database_errors:
File "/usr/local/lib/python3.10/dist-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
self.connect()
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python3.10/dist-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "db" (172.19.0.2), port 5432 failed: FATAL: password authentication failed for user "rengine"
yogeshojha
commented
2 months ago Hi @anuj-kumar1SF @its-mr-monday
If both of your problems are related, I can see password auth failed.
Do you mind checking your .env file? the postgres password should be there. I believe you would have changed it earlier during setup, check it and let me know if that's changed now. We dont have any change on the release.
Also can you tell me how did you update reNgine?
justin-davis-uplight
commented
2 months ago Running into the same issue updated from 2.1.3 to 2.2.0 using the update.sh, make build make up process.
yogeshojha
commented
2 months ago Hi @justin-davis-uplight sorry for the inconvenience. I looked at the code and see that somebody had sent this PR for update.sh earlier and has stashed the .env changes that's why we arent able to connect to postgres with latest changes.
I will fix that in next release by creating a backup of .env file.
For now lets focus on fixing this temporarily.
Can you do git stash pop see if .env changes are back?
justin-davis-uplight
commented
2 months ago My .env looks correct, getting http 500 now. I did a make down and make up
vipul-soman
commented
2 months ago Getting same issue , Server Error (500), getting this for sudo make logs
its-mr-monday
commented
2 months ago I was able to manually set the psql password but now I am getting the same Server Error (500) page along with the same logs as above
ncharron
commented
2 months ago It's because the migrations are not being made properly for existing installs. @yogeshojha every time you make a major upgrade (same thing happened from 2.0.7 -->2.10) and now again to 2.2.0, you merge all your migrations into a file called 0001_initial.py. This breaks any existing install. That means we need to do a clean install anytime you do that. Unless we do the changes manually to the db.
If you do not want to break existing installs, you need to leave the migrations intact or else any change to the db you make will break existing installs.
In this particular scenario, it has to do with the user preferences in the dashboard. And I am sure once this is fixed, others will pop up.
yogeshojha
commented
2 months ago @ncharron I wasnt aware of this. Yes I did have to clean up the 0001_initial.py
And maybe that is why I didn't notice this breaking changes on my local instances.
For future releases, I will take that in mind.
can you help with intermediate solution? I am trying to write script that does create the missing table, other than that if you can offer some help would appreciate.
ncharron
commented
2 months ago Yeah absolutely! Well at the moment I would have to find all the differences between the last db and this one. It will be on a case by case basis to be honest depending from which point a person is upgrading.
Let me see what I can do from my end, it would have to be, revert back to where I was for me, import the new models, let django figure out the new migrations, take those migration files and import them into 2.2.0.
yogeshojha
commented
2 months ago Thanks @ncharron!
I will do the fresh 2.1.3 install myself and try to do the update let me see if I can find the differences in migrations.
For this specific case, do you think if I create a script to populate the db manually will it work? For example this
CREATE TABLE IF NOT EXISTS dashboard_userpreferences (
id SERIAL PRIMARY KEY,
bug_bounty_mode BOOLEAN NOT NULL DEFAULT TRUE,
user_id INTEGER UNIQUE NOT NULL REFERENCES auth_user(id) ON DELETE CASCADE
);Yes, you could create a script that would do the changes manually and then it should be fine. However depending on the relationships, you might run into issues with that. But it is worth a shot for now!
yogeshojha
commented
2 months ago Thank you for the heads up, I will try to fix this temporarily now, and in the future, as you said will not touch the initial migrations. I was totally unaware of the consequences that making changes to initial migration will cause this issue.
ncharron
commented
2 months ago So quick way to to do this: Take the 0001_initial.py files from the 2.1.0 repo and put it in a 2.2.0 repo. Then make up on the 2.2.0, django should automatically create all the migrations you need. Currently trying to test it but have data so it's getting tricky :)
yogeshojha
commented
2 months ago @ncharron I have been doing the same thing, I am manually putting back 0001_initial from 2.1.3 and also creating 0002_initial based on the new model
yogeshojha
commented
2 months ago I will be pushing the changes to this branch: https://github.com/yogeshojha/rengine/tree/hotfix/2.2.0_release and will let you know if/when I make enough changes so that some one could try it out.
ncharron
commented
2 months ago So we did manage to fix it using that method. We just had to make a few tweaks on our end because we have a custom model. But in essence, replacing the 0001_initial.py with the previous version, then running makemigrations and the migrate, fixed the issue for the db.
yogeshojha
commented
2 months ago @ncharron Did you test the branch I pushed above? or are you sending PR?
ncharron
commented
2 months ago We haven't been able to test that fix exactly because ours is slightly different because I was still at 2.1.0. But that is the same steps I took to make it work. If someone is coming from 2.1.3, That should resolve the issue AS LONG as the db is still in the 2.1.3 format.
yogeshojha
commented
2 months ago Hi @ncharron @its-mr-monday @vipul-soman @justin-davis-uplight @anuj-kumar1SF
I have fixed this issue in this branch
https://github.com/yogeshojha/rengine/tree/hotfix/2.2.0_release
I have tested it out, upgrading from 2.1.3 to 2.2.0, and it works fine, with no migration issues.
I'm sorry for this silly mistake. @ncharron Thank you for letting me know about the 0001_initial migration. There's always something new to learn, and I hope to rectify this in future releases.
Steps to test this hotfix
git fetch && git checkout hotfix/2.2.0_releaseand do as usual
make down && make build && make up && make logsPlease somebody confirm if it is working and I will merge it
Thanks
sanyam2602
commented
2 months ago @yogeshojha I can confirm it is working.
yogeshojha
commented
1 month ago @sanyam2602 Thank you for testing it out. I have merged the changes. please git pull for latest fixes.
Thanks @ncharron again for the help. Much appreciated
I will keep this issue open for some time to continue the conversation
ncharron
commented
1 month ago No problem!
To clarify for anyone out there if it wasn't clear what was happening:
After every release, the database changes were all merged into 1 file called 0001_initial.py for the different parts of the app. These are created automatically by django based on your model files. These files (0001_initial.py) are found in the migrations folder under different parts of the app. And this was easily done by simply deleting all the files in the migrations folder and letting the app make a clean version of the migrations.
These files are automatically created to note the changes between your Models files and the database structure every time you run the command manage.py makemigrations from the web docker container.
This is useful when developping because if ever you create a new Model for the db or makes changes to a model and therefore to the db, they are automatically picked up and then you can apply them by doing manage.py migrate from the web docker container.
These commands are mostly run automatically based on the entrypoint.sh files for the docker containers every time you start them. So every time you restart your containers it automatically picks up changes to the files, therefore the db and applies them.
And every time you make changes to the models, a new SEQUENTIAL file is created based on those changes to tell the app in what order to apply the migrations. Also all db's managed by django using this method keeps track of which migration you are landed at so it doesn't try to apply all the changes all the time.
You typically end up with different files called 0001_initial.py, 0002_comment_about_changes.py, 0003_something.py, etc.... Now this gets messy after a while but if ever you get rid of all of those and make a 0001_initial.py again, unless that database structure is already where it is supposed to be at, your db will think you are at migration 0003 when all there is in the new release is 0001_initial.py. And so it will not pick all the most recent changes made to the db. Now the app will work as intended until you run into one of the new models or differences in the structure and it tries to call those columns/tables that do not exist. That's how we get the 500 error.
In this case it was the user_preferences in the dashboard section of the app.
TL:DR; If you remove all the migrations from the app to make it cleaner, your old db will not know about the changes being made to the code and will crash when trying to fetch something that doesn't exist. This is ONLY a problem for existing installations and not for new installations.
Hope this helps!
vipul-soman
commented
1 month ago It's working out of the box now with all previous data intact
justin-davis-uplight
commented
1 month ago I think I had my directory content messed up. I ended up backing up my rengine dir, did a fresh clone, copied my .env file and the secrets dir and ran the update.sh script again
yogeshojha
commented
1 month ago @ncharron Once again thank you for the detailed explanation and a lesson for me to never touch 0001_initial and further migration files ;p
Prisoner2-6-7
commented
1 month ago I was also getting bad gateway. Now i wouldnt apply the hotfix. Below is the output
git fetch && git checkout hotfix/2.2.0_release
error: pathspec 'hotfix/2.2.0_release' did not match any file(s) known to git
Prisoner2-6-7
commented
1 month ago ❯ git pull Already up to date.
@sanyam2602 Thank you for testing it out. I have merged the changes. please
git pullfor latest fixes.Thanks @ncharron again for the help. Much appreciated
I will keep this issue open for some time to continue the conversation
❯ git pull Already up to date.
but i am still getting 502 bad gateway.
startScan.EndPoint.techs: (fields.W340) null has no effect on ManyToManyField.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
self.connect()
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python3.10/dist-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "db" (172.25.0.4), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/app/manage.py", line 26, in <module>
main()
File "/usr/src/app/manage.py", line 22, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.10/dist-packages/django/contrib/auth/management/commands/createsuperuser.py", line 79, in execute
return super().execute(*args, **options)
File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 397, in execute
self.check_migrations()
File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 486, in check_migrations
executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/executor.py", line 18, in __init__
self.loader = MigrationLoader(self.connection)
File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/loader.py", line 53, in __init__
self.build_graph()
File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/loader.py", line 220, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/recorder.py", line 77, in applied_migrations
if self.has_table():
File "/usr/local/lib/python3.10/dist-packages/django/db/migrations/recorder.py", line 55, in has_table
with self.connection.cursor() as cursor:
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 259, in cursor
return self._cursor()
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 235, in _cursor
self.ensure_connection()
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 218, in ensure_connection
with self.wrap_database_errors:
File "/usr/local/lib/python3.10/dist-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
self.connect()
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python3.10/dist-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "db" (172.25.0.4), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
make: *** [Makefile:36: username] Error 1
Using: docker-compose
COMPOSE_DOCKER_CLI_BUILD=1 docker-compose -f docker-compose.yml exec web python3 manage.py migrate
_ _ _
| \ | | (_)
_ __ ___| \| | __ _ _ _ __ ___
| '__/ _ \ . ` |/ _` | | '_ \ / _ \
| | | __/ |\ | (_| | | | | | __/
|_| \___|_| \_|\__, |_|_| |_|\___|
__/ |
|___/
pikepdf._core | pikepdf C++ to Python logger bridge initialized
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
self.connect()
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/base/base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.10/dist-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python3.10/dist-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "db" (172.25.0.4), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/app/manage.py", line 26, in <module>
main() ```Okay i had the problem after automatically running in installation process.
I manually sudo make down && sudo make up
and it works now
ncharron
commented
1 month ago Yeah so to be clear your issue was this line:
django.db.utils.OperationalError: connection to server at "db" (172.25.0.4), port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections?
That happens when there is a docker network problem and you have to do sudo make down (to delete the network) and then sudo make up (or just sudo docker compose up -d since sudo make up rebuilds the entire containers) to bring back up the networks and the containers will be able to speak to each other again.
Is there an existing issue for this?
Current Behavior
Rengine was working fine 3 days ago from raising this bug. 2 days ago I updated the tool and since then I'm facing 502 bad gateway. My kali linux is already up to date and have all the dependencies required to run this tool. I've look for any solution but nothing is working. I've tried reinstalling it but still no luck. Please somebody help.
Expected Behavior
It should have worked correctly.
Steps To Reproduce
Install the tool open browser localhost or 127.0.0.1
Environment
Anything else?