Attempt on connectivity-search-manuscript produced aggressive suggestions

[dhimmel]

I just attempted to dispatch the ai-revision workflow for connectivity-search-manuscript in this job.

The revise step logged:

Using language model from environment variable 'AI_EDITOR_LANGUAGE_MODEL'
Language model:  text-davinci-003
File names to revise: {'05.main-text.md'}

But took zero seconds and no output branch or PR was created. @miltondp any idea what happened?

[dhimmel]

Hmm possibly it only works with preset file names?

https://github.com/manubot/manubot-ai-editor/blob/a571f4dde356a212d3e37fc0d6c5a8ff67e5b935/libs/manubot_ai_editor/editor.py#L190-L205

Perhaps there can be a fallback method & prompt? Or log or throw an error in this case?

Looks like I might be able to set an environment variable:

https://github.com/manubot/manubot-ai-editor/blob/a571f4dde356a212d3e37fc0d6c5a8ff67e5b935/libs/manubot_ai_editor/env_vars.py#L32-L44

[dhimmel]

Got it working with these tweaks in https://github.com/dhimmel/connectivity-search-manuscript/commit/6606ed37efc717782ad67531a7c6760c35ceccc8 with a PR at https://github.com/greenelab/connectivity-search-manuscript/pull/48. It went a bit wild with rewriting, so probably won't end up using.

Basically what I'm looking for is:

• copyediting at the entire manuscript scope for typos, grammar issues, misspellings, etcetera.
• shortening of select sections, possibly with custom promps

For the second point, perhaps there's a construct where certain HTML comments in the source markdown can activate chatgpt for specific ranges, with each comment being able to specify a preset or even custom prompt.

[cgreene]

I'm surprised by how aggressive it was with the rewrite. I agree that, in this case, it's going to create more work.

Maybe it was so aggressive because you used the methods prompt? Definitely an interesting case study.

[miltondp]

Hi @dhimmel. Sorry for the late reply. I was out of town. Thank you for your feedback! I'm sorry it didn't work as expected for you. Some suggested changes don't make any sense.

The current prompts are very "hands-on", with specific instructions for different sections, and that's probably influencing this output. Lately, I was trying to use a prompt like "proofread and revise the following paragraph" in ChatGPT and it seems to work well. It might be what you are looking for. However, we don't have yet support for custom prompts in the manubot-ai-editor. I will try to add some basic support and see if it helps. I will let you know.

I also agree that a more verbose output should indicate if no files were found. I created issue #14 for that.

[miltondp]

I'm changing the title of the issue to reflect the problem of aggressive editing.

@dhimmel, I change manubot-ai-editor to support basic custom prompts. The latest version should be automatically installed everywhere now. I also created a PR in your manuscript repo if you want to give it a try. The custom prompt "proofreads and revises" each paragraph, without any section-specific instructions. You can also try to include "proofread" only in the prompt, but the actual one seems to work.

[dhimmel]

I tried and proofread and revise in https://github.com/greenelab/connectivity-search-manuscript/pull/51 / https://github.com/greenelab/connectivity-search-manuscript/pull/51/commits/34f39be6aa4c46a2e5af1bb5b1aebc0e31292994, but the results are too aggressive. I think I should retry with just proofread.

[dhimmel]

The proofread only prompt was less aggressive as noted in https://github.com/greenelab/connectivity-search-manuscript/pull/51#issuecomment-1559206047. It used:

AI_EDITOR_CUSTOM_PROMPT: "Proofread the following paragraph"

Ended up taking several suggestions from it in https://github.com/greenelab/connectivity-search-manuscript/commit/8ad541fbe6791be7cb74c90c62ae8334cb1800e6.

[miltondp]

Thank you for the feedback, @dhimmel!