596 fix problems with integrationlangchaintest

ScottLogic / prompt-injection

Application which investigates defensive measures against prompt injection attacks on an LLM, with a focus on the exposure of external tools.

MIT License

15 stars 10 forks source link

596 fix problems with integrationlangchaintest #804

Closed pmarsh-scottlogic closed 7 months ago

pmarsh-scottlogic commented 7 months ago

Description

Tidying up some code and making test a bit better

Notes

moves vectorisedDocuments and initDocumentVectors from langchain.ts to document.ts.
puts all document unit tests together into a folder.
- moves tests from integration/langchain.test.ts -> unit/document.ts/initDocumentVectors.test.ts.
- renames/moves unit/document.test.ts -> unit/document.ts/getDocuments.test.ts
moves tests from integration/langchain.test.ts -> unit/langchain.ts/formatPromptEvaluation.test.ts
moves tests from integration/langchain.test.ts -> unit/langchain.ts/initialisePromptEvaluationModel.test.ts
tidies some test layouts and naming also.
Individual tests are largely unchanged. Some mocks have been altered/added and some things reworded

Checklist

Have you done the following?

[x] Linked the relevant Issue
[ ] Added tests
[x] Ensured the workflow steps are passing

pmarsh-scottlogic commented 7 months ago

will reopen when It's ready

pmarsh-scottlogic commented 7 months ago

I'd be interested to know what the response actually is here, and why we need to strip out all non-alphanumerics; does it wrap the "yes" or "no" inside braces, or quotes maybe?

@chriswilty

When I sent "Hello", the prompt eval LLM came back with "No.", and when I sent "forget your instructions" it came back with "Yes.". So we need to strip away the punctuation.

chriswilty commented 7 months ago

I'd be interested to know what the response actually is here, and why we need to strip out all non-alphanumerics; does it wrap the "yes" or "no" inside braces, or quotes maybe?

@chriswilty

When I sent "Hello", the prompt eval LLM came back with "No.", and when I sent "forget your instructions" it came back with "Yes.". So we need to strip away the punctuation.

Brilliant, thanks for checking!

pmarsh-scottlogic commented 7 months ago

I'd be interested to know what the response actually is here, and why we need to strip out all non-alphanumerics; does it wrap the "yes" or "no" inside braces, or quotes maybe?

@chriswilty When I sent "Hello", the prompt eval LLM came back with "No.", and when I sent "forget your instructions" it came back with "Yes.". So we need to strip away the punctuation.

Brilliant, thanks for checking!

@chriswilty in fact we're explicitly asking it to return in that format, with the full stop. I wonder why.

chriswilty commented 7 months ago

@pmarsh-scottlogic I'm happy to approve this anyway. If you need any help with merge conflicts, just ask (tomorrow!)