Task: DOC - Document Workflow

[luxedo]

Description:

I have a few questions:

1. What's the important steps to set up this project?
2. Which scripts should I run?
3. Where do I store the examples? Or are they sent directly to the numpy repo
4. How does the review process work?

I think that some documentation about the workflow would help newcomers to get going more easily. An overview in README.md already helps a lot. Some sections like: Installation, Project structure, Running, ..., can help guide a new person into the project quicker.

Acceptance Criteria:

• [ ] Document project setup: installation, build, ...
• [ ] Document an overview of the project structure: What's the responsibility of each directory?
• [ ] Document workflow. Eg:
  1. Find function without documentation
  2. Run script A to generate examples
  3. Commit examples to directory B
  4. Submit for review
  5. Create PR at numpy when done

[bmwoodruff]

Welcome @luxedo, glad you stopped by. We're currently working to figure out answers to all the questions you asked.

Right now, we're still working on an automated workflow. I decided to focus on example generation as the first option, and hope to move on to other things when I've got that done. OpenTeams has contributed a server for a few student interns and I to help generate prompts. We're using Llama3. Otherwise, you'd need your own API key to run the script.

• The current example-generator file has been run on the entire NumPy code base, with output stored in logs, waiting to be processed.
• The output has to be processed. New examples have to be stripped of all "hallucinated" output and replaced with correct output. This can be done with reviewtools.py. This script needs a few more enhancements to make sure proper spacing for NumPy standards is correct, and sometimes the output for single values is not what I'd want (the difference between repr and print)
• We need a script that will insert the new examples into the proper place in the code base (that's what I'm currently working on). See https://github.com/possee-org/genai-numpy/issues/77.

The current review process for the interns is to

• make changes on a branch
• push to their fork,
• submit a review task for a tech lead on this project (currently that's me). We have a few items that have been reviewed, with a lot more coming this week.
• get feedback from @charris - We don't want to inundate the PR list with hundreds or thousands of new examples, until someone on the core team believes the work is useful.
• interns send PR to NumPy.

For someone who isn't an intern, the review process isn't defined, and they could use this tool in whatever way they want. For example, you could use the output logs from Llama3-70B

What I want to do next is to focus on general docstring improvements. We'll need to get a good prompt (I found AI itself can help generate the prompt). I would like to refine prompts to get fewer false positives (the current example script generates a lot of low quality examples, which we will toss).

There are tons of other ways this project could go. Two months ago I knew nothing about unit tests. Now I think I could start asking AI for help with writing unit test to improve coverage.

The devs specifically asked if there were some way that AI could help them identify lower quality PRs in some algorithmic way, and git them a way to help automate away some of the feedback they need to give in such a case. I think it's doable, but my skill set and familiarity with what "low-quality-PR" means is not yet where I want it to be to start such a task.

[luxedo]

Thats a good overview, thanks!

I have some experience with unit tests and CI/CD pipelines, so I can help with that to help automate some verifications for this repo. I can also split into smaller tasks to make it easier to follow.

As for identifying low-quality PRs I think it is possible given the history of closed PRs, but a better model would need some human validation to curate this dataset.

I'm closing this issue. Please let me know if there's any task you think I can be of help.