Feat/haystack integration

[bergr7]

Summary Create haystack integration and new tutorial.

Description

• Haystack integration
• Fix _determine_batch_size in FlowJudgeHFModel
• Haystack tutorial

This has been tested locally.

There are no automated tests for integrations yet. Waiting for unit tests revamp.

[sariola]

note to add dependency

[bergr7]

note to add dependency

It's in a comment at the top of the notebook. Same as in llama index tutorial.

Do you think we need to add it as a dependency to our pyproject.toml?

[sariola]

note to add dependency

It's in a comment at the top of the notebook. Same as in llama index tutorial.

Do you think we need to add it as a dependency to our pyproject.toml?

I understand.

How about something of this sort? There are always people who don't read like me.

try:
    from haystack import Document
except ImportError:
    print("Haystack is not installed. ")
    print("Please install it according to the 'Additional Requirements' section above.")
    print("\nAfter installation, restart the kernel and run this cell again.")
    raise SystemExit("Stopping execution due to missing Haystack dependency.")

[bergr7]

note to add dependency

It's in a comment at the top of the notebook. Same as in llama index tutorial. Do you think we need to add it as a dependency to our pyproject.toml?

I understand.

How about something of this sort? There are always people who don't read like me.

try:
    from haystack import Document
except ImportError:
    print("Haystack is not installed. ")
    print("Please install it according to the 'Additional Requirements' section above.")
    print("\nAfter installation, restart the kernel and run this cell again.")
    raise SystemExit("Stopping execution due to missing Haystack dependency.")

Added suggestion to llama index and haystack tutorials.

[sariola]

Could add that for a finegrained token need to enable these permission scopes

[sariola]

The batch eval at the end takes about 2 min per run and uses multi-gpu ok via accelerate. Works :ok:

[sariola]

I consider it E2E tested from my side!

[sariola]

Does it handle parsing errors for you guys?

[bergr7]

The batch eval at the end takes about 2 min per run and uses multi-gpu ok via accelerate. Works 🆗

yes it's quite slow... hopefully we can replace with vllm engine soon or cpu.

[bergr7]

Does it handle parsing errors for you guys?

flow-judge does, haystack doesn't. I don't know if there is a way of working around them. Need to check.

[bergr7]

Does it handle parsing errors for you guys?

I guess this is due to the phi-3 issue? have you checked the output?

[sariola]

Does it handle parsing errors for you guys?

I guess this is due to the phi-3 issue? have you checked the output?

Yeah, I'm on it. I think the onnxruntime engine just takes up all the VRAM and cranks even on the sync exec. I'm making some changes to the current test implementation of batch_generate of the class. Should be done in a bit.