Add a BT condition to check robot's vicinity with an AI model (currently ChatGPT) - Githubissues

ros-navigation / navigation2

ROS 2 Navigation Framework and System

https://nav2.org/

Other

2.3k stars 1.2k forks source link

Add a BT condition to check robot's vicinity with an AI model (currently ChatGPT) #4486

Closed AndyZe closed 1 week ago

AndyZe commented 1 week ago

Note: I leave for vacation on 6/26, back on 6/30. So I won't be responsive during that time. I think that's good, it will give you time to noodle over how to handle the AI aspect. I'll write some tests when I get back.

PR requirements:

requires an OpenAI account. I believe even a free account will work; I pay $20/month.
you need to set an environment variable like so. Get your key here: https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key

export OPENAI_API_KEY="..."

Basic Info

Info	Please fill out this column
Ticket(s) this addresses	Fixes #4484
Primary OS tested on	Ubuntu
Robotic platform tested on	TODO, RealSense D415
Does this PR contain AI generated software?	No

Description of contribution in a few bullet points

Sends a request out to ChatGPT to look for anything blocking forward progress in the robot's vicinity
Uses one camera image as input
Useful for detecting things like cords on the ground, cracks, pools of water, small animals, hot lava ... etc... that Lidar and pointclouds won't
It's a fuzzy check. It's not parameterized. Basically it uses the AI model's "common sense"

chatgpt

pool

Description of documentation updates required from your changes

Need to update the BT plugin tutorials

Future work that may be required in bullet points

I think there will be a need to call multiple AI models in the near future. It won't just be ChatGPT. Might want to think about a separate library/interface for calling/prompting models

For Maintainers:

[ ] Check that any new parameters added are updated in docs.nav2.org
[ ] Check that any significant change is added to the migration guide
[ ] Check that any new features OR changes to existing behaviors are reflected in the tuning guide
[ ] Check that any new functions have Doxygen added
[ ] Check that any new features have test coverage
[ ] Check that any new plugins is added to the plugins page
[ ] If BT Node, Additionally: add to BT's XML index of nodes for groot, BT package's readme table, and BT library lists

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

mergify[bot] commented 1 week ago

@AndyZe, your PR has failed to build. Please check CI outputs and resolve issues. You may need to rebase or pull in main due to API changes (or your contribution genuinely fails).

SteveMacenski commented 1 week ago

I'm going to again ask that this is closed and to have a fuller conversation before asking for reviews. See ticket you filed #4484

Please do not use our CI in place of your own development environment. We have limited build time and CI is not the place to be fixing your basic build issues.

Note: I leave for vacation on 6/26, back on 6/30

Given this, I'm going to close this for you since this shouldn't sit on our queue while you're gone

AndyZe commented 1 week ago

Why so grumpy, Steve? I don't want it to be like that.

If you let me target Humble branch, I would be able to develop locally much easier. Rolling is an inconvenience.

SteveMacenski commented 1 week ago

Why so grumpy, Steve? I don't want it to be like that.

Sorry, not grumpy, this is boilerplate text, I get students frequently that submit broken PRs and trigger dozens of jobs to get it to build since they're not attempting to build locally and it just sucks up all of our build time and makes things hard to review.

In general though, I expect that PRs are in a complete I-want-a-review-to-merge state. If we're still talking more 'how should we do this' and 'does something like this belong in Nav2' kind of stuff, a ticket for discussion with perhaps a link a branch that is in progress to poke around is better process IMO. Carpet bombing maintainers with emails and notifications about broken builds in a PR doesn't look good :laughing:

If you let me target Humble branch, I would be able to develop locally much easier. Rolling is an inconvenience.

You can still develop in Humble and then forward port to Rolling once ready. Its not like these should vary significantly between them, should be a sub-10 minute forward port. You could dev on humble locally until its ready for a review, then go to Rolling.