search

Azure / PyRIT

The Python Risk Identification Tool for generative AI (PyRIT) is an open access automation framework to empower security professionals and machine learning engineers to proactively find risks in their generative AI systems.
MIT License
1.82k stars 351 forks source link

DOC Testing PyRIT Locally #480

Open Tiger-Du opened 3 days ago

Tiger-Du commented 3 days ago

Describe the issue linked to the documentation

Hello! While running the tests for PyRIT locally with make test-cov-xml, I came across a few errors that resulted from aria2, pyodbc, and unixODBC not being installed locally. These errors do not appear in Actions since the ubuntu-latest image has these packages installed by default from what I believe (https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md).

Suggest a potential alternative/fix

Perhaps we should add a note somewhere in the doc/contributing folder that Ubuntu is the intended environment for testing at the moment and that aria2, pyodbc, and unixODBC should be installed. I'd be happy to do so but would love to hear what everyone thinks first!

romanlutz commented 2 days ago

Are you using MacOS? Most of us use Windows and Ubuntu is used in the pipelines so they tend to work best.

We should probably try and set up macOS pipelines at some point.

Feel free to add extra info for macOS if I am indeed making correct assumptions.

Tiger-Du commented 2 days ago

Yep, I am indeed using macOS. And okay, great, I'll go ahead and open a PR to add some more information.

As far as setting up macOS pipelines, I think that it would require changing os: [ubuntu-latest] in L25 of .github/workflows/build_and_test.yml to os: [ubuntu-latest, macos-latest]. However, the runs take around 10 minutes at the moment, and adding macOS would double the time, which might be inconvenient.

romanlutz commented 1 day ago

They run in parallel, shouldn't be a problem. The setup usually doesn't "just work", though. So there may be some work in getting the setup working on the build agent. You can give it a try if you like!