how to add new embedding models?

[LexiestLeszek]

Hi!

I thought about adding couple more embedding models and wanted to do that via Exporters tool to convert them from HF to CoreML (Zach, I assume you did it via that tool also), but couldnt figure out how make the resulted CoreML model not in the size of default 128 tokens for both input and output (the Exporters library seems to be having this size as a default and every model that you put through Exporters results in having this size as input and output).

Any ideas how to make that happen?

[ZachNagengast]

Hi @LexiestLeszek more models would be amazing! I've been trying to keep them under the 100mb limit for github but I think it may be better to just setup a downloader or customer path loading here. Also, the new multilanguage model from Apple seems pretty promising https://developer.apple.com/documentation/naturallanguage/nlcontextualembedding.

I just saw this Exporters app, probably would have saved me some time but I just used this script for the conversion: https://gist.github.com/ZachNagengast/ce84f1056835987a7678558b3fc6c69d.

I noticed in that app, you can adjust the input token length with a change like this:

class MyCoreMLConfig(DistilBertCoreMLConfig):
    @property
    def inputs(self) -> OrderedDict[str, InputDescription]:
        input_descs = super().inputs
        input_descs["input_ids"].sequence_length = 512 <--- change this number
        return input_descs

Worth noting that the input token length has a pretty big impact on performance depending on the model. Anecdotally, I've tried using flexible input shapes with the distilbert conversion and it slowed it down quite a bit.

[LexiestLeszek]

I converted one of the models (bigger bert model, sentence-transformers/msmarco-bert-base-dot-v5), copied and changed a swift class from distilbert model like this:

//
//  BertEmbeddings.swift
//  
//
//  Created by Leszek Mielnikow on 27/07/2023.
//

import Foundation
import CoreML
import SimilaritySearchKit

@available(macOS 13.0, iOS 16.0, *)
public class BertEmbeddings: EmbeddingsProtocol {
    public let model: msmarco_bert
    public let tokenizer: BertTokenizer
    public let inputDimention: Int = 512
    public let outputDimention: Int = 768

    public init() {
        let modelConfig = MLModelConfiguration()
        modelConfig.computeUnits = .all

        do {
            self.model = try msmarco_bert(configuration: modelConfig)
        } catch {
            fatalError("Failed to load the Core ML model. Error: \(error.localizedDescription)")
        }

        self.tokenizer = BertTokenizer()
    }

    // MARK: - Dense Embeddings

    public func encode(sentence: String) async -> [Float]? {
        // Encode input text as bert tokens
        let inputTokens = tokenizer.buildModelTokens(sentence: sentence)
        let (inputIds, attentionMask) = tokenizer.buildModelInputs(from: inputTokens)

        // Send tokens through the MLModel
        let embeddings = generateBertEmbeddings(inputIds: inputIds, attentionMask: attentionMask)

        return embeddings
    }

    public func generateBertEmbeddings(inputIds: MLMultiArray, attentionMask: MLMultiArray) -> [Float]? {
        let inputFeatures = msmarco_bertInput(
            input_ids: inputIds,
            attention_mask: attentionMask
        )

        let output = try? model.prediction(input: inputFeatures)

        guard let embeddings = output?.embeddings else {
            return nil
        }

        let embeddingsArray: [Float] = (0..<embeddings.count).map { Float(embeddings[$0].floatValue) }

        return embeddingsArray
    }
}

and see an error "Property cannot be declared public because its type uses an internal type" next to "public let model: msmarco_bert".

How to fix this error?

[LexiestLeszek]

Any ideas? :)

[ZachNagengast]

I'm not exactly sure without seeing the model, but the class is automatically generated, so there may be an issue with that. You could try checking the class definition and see if anything looks off?

[LexiestLeszek]

Oh, no, i didn't use automatically generated class, I copied your code from DistilbertEmbeddings and changed the names of variables. Is it the problem? I still don't quite understand how classes of mlmodels in CoreML work...

[ZachNagengast]

Oh ok that could be the issue, the class name of the model will show up in code as the exact name of the .mlpackage file. So if you start typing the name of the file, there should be an autocomplete if it has been auto-generated. If it's not autogenerating there may be some other issue.

[ZachNagengast]

Copying and pasting the EmbeddingsProtocol class is ok, but the line

public let model: msmarco_bert

needs to be specifically the same as the model's file name.

[LexiestLeszek]

yes, i also found out that if I use the models from the package, like DIstilBert for example, i can click on this and it will open a swift code:

while in my model nothing happens when I click on it... Any ideas?

[LexiestLeszek]

Ok so it needed to be built. Just clicking Build solved the thing with not appearing swift class.

But the issue with "Property cannot be declared public because its type uses an internal type" next to "public let model: msmarco_bert" is still there. Maybe I am messing up with classes? BertEmbeddings (the swift file i created) is in the root directory, maybe i need to add it to the package?

[LexiestLeszek]

Problem identified! For some reason, the swift class that mlpackage created automatically for my new model is not public.

This is the piece of code from swift class of DistilbertEmbeddings model from the SimilaritySearchKit package:

/// Class for model loading and prediction
@available(macOS 13.0, iOS 16.0, tvOS 16.0, watchOS 9.0, *)
public class msmarco_distilbert_base_tas_b_512_single_quantized {
    public let model: MLModel

    /// URL of model assuming it was installed in the same bundle as this class
    class var urlOfModelInThisBundle : URL {
        let bundle = Bundle.module
        return bundle.url(forResource: "msmarco_distilbert_base_tas_b_512_single_quantized", withExtension:"mlmodelc")!
    }

And this is the code that my swift class has in msmarco_bert:

/// Class for model loading and prediction
@available(macOS 13.0, iOS 16.0, tvOS 16.0, watchOS 9.0, *)
class msmarco_bert {
    let model: MLModel

    /// URL of model assuming it was installed in the same bundle as this class
    class var urlOfModelInThisBundle : URL {
        let bundle = Bundle(for: self)
        return bundle.url(forResource: "msmarco_bert", withExtension:"mlmodelc")!
    }

The difference is that my class for some reason is not public, as any other classes that are in the swift file for my model

Any ideas on how to fix it?

[LexiestLeszek]

Solved. Sorry for making a blog out of this issue.

In the File Inspector panel on the right side, under "Target Membership," make sure the checkbox next to your app target is checked. This will make the Swift class file accessible to your app.

But now I have the problem that model returns only first chunk of text all the time...

[LexiestLeszek]

@ZachNagengast Do I need to somehow reference the new model in the SimilarityIndex or any other package files?

It doesn't seem to work and .score shows "nan". Meaning similarity index doesn't return anything (in my case it always returned first chunk doesn't matter what query was).

[ZachNagengast]

Yea if you add a new embedding, you'll need to add a new target to the Package.swift, so that people can optionally enable it. Target membership and the Build phases tab are all good places to check.

[LexiestLeszek]

Got it, almost done. In the SimilarityIndex.swift there is this piece:

    /// An enumeration of available embedding models.
    public enum EmbeddingModelType {
        /// DistilBERT, a small version of BERT model fine tuned for questing-answering.
        case distilbert

        /// MiniLM All, a smaller but faster model.
        case minilmAll

        /// Multi-QA MiniLM, a fast model fine-tuned for question-answering tasks.
        case minilmMultiQA

        /// A native model provided by Apple's NaturalLanguage library.
        case native
    }

Where are cases defined? How can i add my model there? when i search for lowercased "distilbert" like in one of the cases, i couldn't find anything. I need to add my model as one more case there i guess, so how do i do it?

[LexiestLeszek]

I installed the package locally but still whenever I choose DistilBert it works perfectly fine, but when I choose new model it just returns first 3 chunks of the text and the .score metric is "nan" (not a number i assume)...

[LexiestLeszek]

It doesn't have any errors with targets, I added everything that is needed to the Package.swift and added target dependencies. PDFExample app creates embeddings using my new model but the search still doesn't work, it always returns Top 5 results ad the First 5 results.

[ZachNagengast]

@LexiestLeszek If you have a branch I can checkout perhaps I can help resolve it like that, otherwise there could be any number of issues going on

[LexiestLeszek]

@ZachNagengast https://github.com/LexiestLeszek/similarity-search-kit

I basically updated:

1. Package.swift to include new target;
2. SimilarityIndex.swift (added enum for my new embeddings)
3. Added a folder to the Addons-> Embeddings called "My Embeddings", where the MyEmbeddings.swift code is copied from DistilbertEmbeddings.swift with changed names of some functions for convenience.

There is no model in the branch due to the size. The msmarco_bert model is a sentence-transformers/msmarco-bert-base-dot-v5 model that I put through your conversion script.

In PDFExample I have this error: Results: [SimilaritySearchKit.SimilarityIndex.SearchResult(id: "id0", score: nan, text: "EMPLOYMENT AGREEMENT\nThis Employment Agreement (the “Agreement”) is effective as of January 6, 2004 (the “Effective Date”) by and between John O’Keefe (the “Executive”) and Verdisys", metadata: ["source": "employment_agreement.pdf"])]

the problem is score: nan It always returns first text chunk doesn't matter what type of question I ask. Note that the model seems to be creating the embeddings just fine, but when it comes to search it doesn't work for some reason.

[ZachNagengast]

Thanks, thats helpful - I tested out your same code got it working. Check out this diff for contentview.swift

The code in your repo is trying to cast MyEmbeddings as DistilbertEmbeddings, which may have been causing the issues. I also uploaded all the converted models here for reference.

https://huggingface.co/ZachNagengast/similarity-search-coreml-models/tree/main

[LexiestLeszek]

Hm, I just realized my code had MyEmbeddings everywhere and i just didn't pushed the file of PDFExample.

Why your ContentView diff has both MyEmbeddings and embeddingModel? What is embeddingModel?

Are you sure it works for you and it doesn't return the id:0 text chunk?

Sorry 😟

[ZachNagengast]

Yea, it works for me - embeddingModel is just a reference to index.indexModel, which is what's used to do the encoding.

If you want to go deep into the workings of it, I'd recommend checking out the vector math in this function step by step - https://github.com/ZachNagengast/similarity-search-kit/blob/0129d90cb49cab6a6d21fcfcfc7c14b2efacdbe9/Sources/SimilaritySearchKit/Core/Embeddings/Metrics/DistanceMetrics.swift#L15-L28

Reasons it could be NaN:

• If firstEmbedding or secondEmbedding contain NaN values.
• If both arrays contain Infinity or -Infinity at the same positions
• If one array is empty, the function returns 0, potentially causing unexpected results.

[LexiestLeszek]

@ZachNagengast i think without "as? MyEmbeddings" it will use NativeEmbeddings as default, not the new model, because SimilarityIndex.swift has this:

    public init(name: String? = nil, model: (any EmbeddingsProtocol)? = nil, metric: (any DistanceMetricProtocol)? = nil, vectorStore: (any VectorStoreProtocol)? = nil) async {
        // Setup index with defaults
        self.indexName = name ?? "SimilaritySearchKitIndex"
        self.indexModel = model ?? NativeEmbeddings()
        self.indexMetric = metric ?? CosineSimilarity()
        self.vectorStore = vectorStore ?? JsonStore()

        // Run the model once to discover dimention size
        await setupDimension()
    }

My PDFExample has this and still doesn't work:

    func loadIndex() {
        Task {
            similarityIndex = await SimilarityIndex(name: "PDFIndex", model: MyEmbeddings(), metric: DotProduct())
        }
    }

                if let miniqa = index.indexModel as? MyEmbeddings {
                for chunk in chunks {
                    if let embedding = await miniqa.encode(sentence: chunk) {
                        embeddings.append(embedding)
                    }
                }
            }

Here is the updated file: https://github.com/LexiestLeszek/similarity-search-kit/blob/main/Examples/PDFExample/PDFExample/ContentView.swift

Everything works perfectly with DistilbertEmbeddings or other pre-packaged models, but with mine - doesnt.

[ZachNagengast]

It all seems fine to me, I can't really figure out the underlying issue without seeing the full implementation, sorry about that. Can you post the code you're using to actually do the search? If the embeddings are being generated then we might be looking in the wrong place for the bug.

[LexiestLeszek]

I am using PDFExample, here is the code of PDF example:

//
//  ContentView.swift
//  PDFExample
//
//  Created by Zach Nagengast on 5/2/23.
//

import SwiftUI
import SimilaritySearchKit
import SimilaritySearchKitDistilbert
import SimilaritySearchKitMyEmbeddings
import UIKit
import MobileCoreServices
import PDFKit
import QuickLookThumbnailing

struct ContentView: View {
    @State private var documentText: String = ""
    @State private var fileName: String = ""
    @State private var fileIcon: UIImage? = nil
    @State private var totalCharacters: Int = 0
    @State private var totalTokens: Int = 0
    @State private var progress: Double = 0
    @State private var chunks: [String] = []
    @State private var embeddings: [[Float]] = []
    @State private var searchText: String = ""
    @State private var searchResults: [String] = []
    @State private var isLoading: Bool = false

    @State private var similarityIndex: SimilarityIndex?

    var body: some View {
        VStack {
            Text("🔍 PDF Search")
                .font(.largeTitle)
                .bold()
                .padding()

            Button(action: selectFromFiles) {
                Text("📂 Select PDF to Search")
                    .font(.headline)
                    .foregroundColor(.white)
                    .frame(maxWidth: 500)
                    .padding()
                    .background(Color.blue)
                    .cornerRadius(8)
            }
            .padding()

            if !fileName.isEmpty {
                HStack {
                    if let fileIcon = fileIcon {
                        Image(uiImage: fileIcon)
                            .resizable()
                            .scaledToFit()
                            .frame(width: 60, height: 60)
                            .cornerRadius(8)
                    }

                    VStack(alignment: .leading) {
                        Text("File: \(fileName)")
                            .font(.headline)
                        Text("🔡 Total Tokens: \(totalTokens)")
                    }
                }
                .padding()

                Button("🤖 Create Embedding Vectors") {
                    vectorizeChunks()
                }
                .font(.headline)
                .foregroundColor(.white)
                .frame(maxWidth: 500)
                .padding()
                .background(Color.blue)
                .cornerRadius(8)
                .padding()
            }

            if !embeddings.isEmpty {
                Text("🔢 Total Embeddings: \(embeddings.count)")
                    .font(.headline)
                    .padding()

                if embeddings.count != chunks.count {
                    ProgressView(value: Double(embeddings.count), total: Double(chunks.count))
                        .frame(height: 10)
                        .frame(maxWidth: 500)
                        .padding()
                } else {
                    TextField("🔍 Search document", text: $searchText, onCommit: searchDocument)
                        .textFieldStyle(RoundedBorderTextFieldStyle())
                        .padding()
                        .frame(maxWidth: 500)

                    List(searchResults, id: \.self) { result in
                        Text(result)
                    }
                    .frame(maxWidth: 500)
                    HStack {
                        Button("📋 Copy LLM Prompt") {
                            exportForLLM()
                        }
                        .font(.headline)
                        .foregroundColor(.white)
                        .frame(maxWidth: 250)
                        .padding()
                        .background(Color.blue)
                        .cornerRadius(8)
                        .padding()

                        Button("💾 Save for Pinecone") {
                            exportForPinecone()
                        }
                        .font(.headline)
                        .foregroundColor(.white)
                        .frame(maxWidth: 250)
                        .padding()
                        .background(Color.blue)
                        .cornerRadius(8)
                        .padding()
                    }
                }
            }
            Spacer()
        }
        .onAppear {
            loadIndex()
        }
    }

    func loadIndex() {
        Task {
            similarityIndex = await SimilarityIndex(name: "PDFIndex", model: MyEmbeddings(), metric: DotProduct())
        }
    }

    func selectFromFiles() {
        let picker = DocumentPicker(document: $documentText, fileName: $fileName, fileIcon: $fileIcon, totalCharacters: $totalCharacters, totalTokens: $totalTokens)
        let hostingController = UIHostingController(rootView: picker)
        UIApplication.shared.connectedScenes
            .map { ($0 as? UIWindowScene)?.windows.first?.rootViewController }
            .compactMap { $0 }
            .first?
            .present(hostingController, animated: true, completion: nil)
    }

    func vectorizeChunks() {
        guard let index = similarityIndex else { return }

        Task {
            let splitter = RecursiveTokenSplitter(withTokenizer: BertTokenizer())
            let (splitText, _) = splitter.split(text: documentText)
            chunks = splitText

            embeddings = []
            if let miniqa = index.indexModel as? MyEmbeddings {
                for chunk in chunks {
                    if let embedding = await miniqa.encode(sentence: chunk) {
                        embeddings.append(embedding)
                    }
                }
            }

            for (idx, chunk) in chunks.enumerated() {
                await index.addItem(id: "id\(idx)", text: chunk, metadata: ["source": fileName], embedding: embeddings[idx])
            }
        }
    }

    func searchDocument() {
        guard let index = similarityIndex else { return }

        Task {
            let results = await index.search(searchText)

            print("Question: \(searchText)")
            print("Results: \(results)")

            searchResults = results.map { $0.text }
        }
    }

    func exportForLLM() {
        guard let index = similarityIndex else { return }

        Task {
            let results = await index.search(searchText, top: 6)
            let llmPrompt = SimilarityIndex.exportLLMPrompt(query: searchText, results: results)
            let pasteboard = UIPasteboard.general
            pasteboard.string = llmPrompt
        }
    }

    func exportForPinecone() {
        struct PineconeExport: Codable {
            let vectors: [PineconeIndexItem]
        }

        struct PineconeIndexItem: Codable {
            let id: String
            let metadata: [String: String]
            let values: [Float]
        }

        guard let index = similarityIndex else { return }

        let encoder = JSONEncoder()
        encoder.outputFormatting = .prettyPrinted

        // Map items into Pinecone import structure
        var pineconeIndexItems: [PineconeIndexItem] = []
        for item in index.indexItems {
            let pineconeItem = PineconeIndexItem(
                id: item.id,
                metadata: [
                    "text": item.text,
                    "source": item.metadata["source"] ?? "",
                ],
                values: item.embedding
            )
            pineconeIndexItems.append(pineconeItem)
        }

        let pineconeExport = PineconeExport(vectors: pineconeIndexItems)

        do {
            let data = try encoder.encode(pineconeExport)
            let fileName = "\(index.indexName)_\(String(describing: index.indexModel))_\(index.dimension).json"

            if let fileURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.appendingPathComponent(fileName) {
                try data.write(to: fileURL)

                let documentPicker = UIDocumentPickerViewController(forExporting: [fileURL], asCopy: true)
                documentPicker.modalPresentationStyle = .fullScreen

                UIApplication.shared.connectedScenes
                    .map { ($0 as? UIWindowScene)?.windows.first?.rootViewController }
                    .compactMap { $0 }
                    .first?
                    .present(documentPicker, animated: true, completion: nil)
            }
        } catch {
            print("Error encoding index:", error)
        }
    }
}

// MARK: - Document Picker

struct DocumentPicker: UIViewControllerRepresentable {
    @Binding var document: String
    @Binding var fileName: String
    @Binding var fileIcon: UIImage?
    @Binding var totalCharacters: Int
    @Binding var totalTokens: Int

    func makeCoordinator() -> Coordinator {
        Coordinator(self)
    }

    func makeUIViewController(context: Context) -> UIDocumentPickerViewController {
        let picker = UIDocumentPickerViewController(forOpeningContentTypes: [.pdf])
        picker.delegate = context.coordinator
        picker.shouldShowFileExtensions = true
        return picker
    }

    func updateUIViewController(_: UIDocumentPickerViewController, context _: Context) {}

    class Coordinator: NSObject, UIDocumentPickerDelegate {
        var parent: DocumentPicker

        init(_ parent: DocumentPicker) {
            self.parent = parent
        }

        func documentPicker(_: UIDocumentPickerViewController, didPickDocumentsAt urls: [URL]) {
            guard let url = urls.first, let _ = PDFDocument(url: url) else { return }
            let pdfText = Files.extractTextFromPDF(url: url) ?? ""

            parent.document = pdfText
            parent.fileName = url.lastPathComponent
            parent.totalCharacters = pdfText.count
            parent.totalTokens = BertTokenizer().tokenize(text: pdfText).count

            // Create the thumbnail
            let size: CGSize = CGSize(width: 60, height: 60)
            let scale = UIScreen.main.scale
            let request = QLThumbnailGenerator.Request(fileAt: url, size: size, scale: scale, representationTypes: .all)
            let generator = QLThumbnailGenerator.shared
            generator.generateRepresentations(for: request) { thumbnail, _, error in
                DispatchQueue.main.async {
                    guard thumbnail?.uiImage != nil, error == nil else { return }
                    self.parent.fileIcon = thumbnail?.uiImage
                }
            }
        }
    }
}

// MARK: - Previews

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}

[LexiestLeszek]

@ZachNagengast the results.score is always nan and the search always returns first k chunks instead of top k chunks (meaning it doesn't really search, just returns first chunks from the list)

[ZachNagengast]

Yea the code looks like it should be working. Maybe something went wrong with the CoreML conversion - try downloading the model from here and using that. Also try to put a breakpoint in here and make sure the vector outputs look ok:

            embeddings = []
            if let miniqa = index.indexModel as? MyEmbeddings {
                for chunk in chunks {
                    if let embedding = await miniqa.encode(sentence: chunk) {
                        embeddings.append(embedding) //<--breakpoint
                    }
                }
            }

[LexiestLeszek]

I used the model you uploaded :(

results always starts with: (id: "id0", score: nan,

[ZachNagengast]

How do the embedding vectors look?

[LexiestLeszek]

Sentence: What is employee responsibilities? Input Tokens: [101, 2054, 2003, 7904, 10198, 1029, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] Input IDs: Int32 1 × 512 matrix [101,2054,2003,7904,10198,1029,102,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0] Attention Mask: Int32 1 × 512 matrix [1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0] Question: What is employee responsibilities? Results: [SimilaritySearchKit.SimilarityIndex.SearchResult(id: "id0", score: nan, text: "EMPLOYMENT AGREEMENT\nThis Employment Agreement (the “Agreement”) is effective as of January 6, 2004 (the “Effective Date”) by and between John O’Keefe (the “Executive”) and Verdisys, Inc., a California corporation (the “Company”).\n1. Duties and Scope of Employment.\n(a) Position. For the term of this Agreement, the Company agrees to employ the Executive in the position of Executive Vice President and Chief Financial Officer (the “Employment”). The duties and responsibilities of Executive shall include the duties and responsibilities for the Executive’s corporate office and position as set forth in the Company’s bylaws and such other duties and responsibilities as the Company’s Chief Executive Officer and/or Board of Directors may from time to time reasonably assign to the Executive.\n(b) Obligations to the Company. During his Employment, the Executive shall devote his full business efforts and time to the Company. During his Employment, without prior written approval from the Company’s Chief Executive Officer, the Executive shall not render services in any capacity to any other person or entity and shall not act as a sole proprietor or partner of any other person or entity or as a shareholder or other owner owning more than ten percent of the stock or other interests of any other corporation or entity. This obligation, however, shall not preclude Executive from engaging in appropriate civic, charitable or religious activities or from devoting a reasonable amount of time to private investments or from serving on the boards of directors of companies, including closely held companies which are controlled by Executive as long as these activities or services do not materially interfere or conflict with Executive’s responsibilities to, or ability to perform his duties of employment by, the Company under this Agreement. The Executive shall comply with the Company’s policies and rules as they may be in effect from time to time during his Employment.\n", metadata: ["source": "employment_agreement.pdf"]), SimilaritySearchKit.SimilarityIndex.SearchResult(id: "id1", score: nan, text: "(c) No Conflicting Obligations. The Executive represents and warrants to the Company that he is under no obligations or commitments, whether contractual or otherwise, that are inconsistent with his obligations under this Agreement. The Executive represents and warrants that he will not use or disclose, in connection with his employment by the Company, any trade secrets or other proprietary information or intellectual property in which the Executive or any other person has any right, title or interest and that his employment by the Company as contemplated by this Agreement will not infringe or violate the rights of any other person. The Executive represents and warrants to the Company that he has returned all property and confidential information belonging to any prior employer.\n2. Cash and Incentive Compensation.\n(a) Salary. The Company shall pay the Executive as compensation for his services during the first twelve (12) months of his Employment a base salary at a gross annual rate of $175,000. Such salary shall be payable in accordance with the Company’s standard payroll procedures. Only in the event that the parties extend the term of this Agreement pursuant to Section 4(a), then Company shall pay the Executive as compensation for his service during the second twelve (12) months of his Employment a base salary at a gross annual rate of $195,000 and, if applicable, for his service during the third twelve (12) months of his Employment a base salary at a gross annual rate of $215,000. (The annual compensation specified in this Subsection (a), together with any increases in such compensation as a result of an extension of the term of this Agreement pursuant to Section 4(a), is referred to in this Agreement as “Base Compensation.”)\n(b) Bonus. Executive shall receive a one time payment of $40,000 as a sign on bonus and shall be included in the Company Executive Compensation program, whereby senior management are eligible to receive annual bonuses of up to 50% of their base compensation.\n(c) Options. Executive shall be eligible to be considered for stock option grants under the Company’s annual stock option award program as administered by, and at the discretion of, the Compensation Committee of the Board of Directors. An initial option shall be granted to the Executive for the right to purchase an aggregate of 80,000 shares of the Company’s common stock at an exercise price based upon the date of approval by the Compensation Committee of the Board of Directors. Such option shall vest quarterly over the initial term of this Agreement.\n", metadata: ["source": "employment_agreement.pdf"]), SimilaritySearchKit.SimilarityIndex.SearchResult(id: "id2", score: nan, text: "(d) Insurance Coverage Reimbursement. Company agrees to pay Executive at current rate of $394 per month to cover and to be in lieu of medical benefits, as selected by Executive in his sole discretion. Executive understands and agrees that Company is not required to permit Executive to participate in any Company-sponsored benefit plans, including but not limited to the Company’s medical plan, in the same or any manner as Company and any third-party benefit provider make such opportunities available to Company’s regular full-time employees; provided, however, if Company does provide such Company-sponsored benefit plans and Executive is eligible to participate in such Company-sponsored benefit plans, Executive shall not be entitled to receive such $394 monthly payment.\n(e) Vacation. During the term of this Agreement, Executive shall be entitled to vacation each year in accordance with the Company’s policies in effect from time to time, but in no event less than four (4) weeks paid vacation per calendar year.\n3. Business Expenses. During his Employment, the Executive shall be authorized to incur necessary and reasonable travel, entertainment and other business expenses in connection with his duties hereunder. The Company shall reimburse the Executive for such expenses upon presentation of an itemized account and appropriate supporting documentation, all in accordance with the Company’s generally applicable\n2\npolicies. The Executive shall have a car allowance of $1,000.00 per month during the term of this Agreement.\n4. Term of Employment.\n(a) Term. This Agreement shall expire on the first anniversary of the Effective Date, unless otherwise extended by the mutual agreement of Executive and the Company; provided, that this Agreement shall automatically be renewed for additional one (1) year terms and shall automatically be continued effective as of the subsequent anniversary date of the Agreement (a “Renewal Date”) unless the Company or Executive has delivered written notice of non-renewal to the other party at least sixty (60) days prior to the relevant Renewal Date.\n", metadata: ["source": "employment_agreement.pdf"]), SimilaritySearchKit.SimilarityIndex.SearchResult(id: "id3", score: nan, text: "(b) Basic Rule. The Executive’s Employment with the Company shall be “at will,” meaning that either the Executive or the Company shall be entitled to terminate the Executive’s Employment at any time and for any reason, with or without Cause (in the case of the Company) or Constructive Termination (in the case of Executive). Any contrary representations that may have been made to the Executive shall be superseded by this Agreement. This Agreement shall constitute the full and complete agreement between the Executive and the Company on the “at will” nature of the Executive’s Employment, which may be changed only in an express written agreement signed by the Executive and a duly authorized officer of the Company.\n(c) Termination. The Company or the Executive may terminate the Executive’s Employment at any time and for any reason (or no reason), and with or without Cause (in the case of the Company) or Constructive Termination (in the case of Executive), by giving the other party notice in writing. The Executive’s Employment shall terminate automatically in the event of his death.\n(d) Rights Upon Termination. Except as expressly provided in Section 5, upon the termination of the Executive’s Employment pursuant to this Section 4, the Executive shall be entitled only to the compensation, benefits and reimbursements described in Sections 2 and 3 for the period preceding the effective date of the termination.\n5. Termination Benefits.\n(a) General Release. Any other provision of this Agreement notwithstanding, Subsections (b), (c), and (d) below shall not apply unless the Employee (i) has executed a general release (in a form reasonably prescribed by the Company) of all known and unknown claims that he may then have against the Company or persons affiliated with the Company, and (ii) has agreed not to prosecute any legal action or other proceeding based upon any of such claims.\n(b) Severance Pay. If, during the term of this Agreement, the Company terminates the Executive’s Employment for any reason other than Cause or Disability, or if the Executive voluntarily resigns following a Constructive Termination, (collectively, a “Termination Event”), then the Company shall pay the Executive his Base Compensation\n 3\n", metadata: ["source": "employment_agreement.pdf"]), SimilaritySearchKit.SimilarityIndex.SearchResult(id: "id4", score: nan, text: "for the remaining period of the then-current term of this Agreement, but not in excess of six (6) months. Such Base Compensation shall be paid as a lump sum within thirty (30) days after the Termination Event. In addition, the Company shall also reimburse the Executive for the payment of the Executive’s COBRA or equivalent other replacement medical and dental insurance premiums for a period of six (6) months.\n(c) Stock Options. With respect to all stock options in the Company granted to Executive prior to the effective date of the Termination Event, an amount of option shares equal to the number of months for which Executive is entitled to receive severance pay will become immediately vested, not subject to repurchase and not subject to assumption, assignment or replacement by the Company or its successors.\n(d) Disability. If the Executive’s employment is terminated by the Company by reason of the Executive’s Disability, the Executive shall be entitled to a prompt cash payment of a prorated portion of the payments set forth in Section 2(a) above for the year in which such termination occurs. Executive and his eligible dependents shall be entitled to continued participation so long as he is disabled and is not eligible for coverage under a successor employer’s plans through the month in which the Executive attains age sixty- five (65) in all medical, dental, vision and hospitalization insurance coverage, and in all other employee welfare benefit plans, programs and arrangements in which he was participating on the date of termination of his employment for Disability on terms and conditions that are no less favorable than those applicable, from time to time, to senior executives of the Company. For purposes of this Agreement, “Disability” means the Executive’s inability, due to physical or mental incapacity, to substantially perform his duties and responsibilities contemplated by this Agreement. In the event of a dispute as to whether the Executive is disabled, the determination shall be made by a licensed medical doctor selected by the Company and agreed to by the Executive. If the parties cannot agree on a medical doctor, each party shall select a medical doctor and the two doctors shall select a third who shall be the approved medical doctor for this purpose. The Executive agrees to submit to such tests and examinations as such medical doctor shall deem appropriate.\n(e) Definition of “Cause.” For all purposes under this Agreement, “Cause” shall mean:\n", metadata: ["source": "employment_agreement.pdf"])]

[LexiestLeszek]

I added this print statements in MyEmbeddings.swift

public func encode(sentence: String) async -> [Float]? {
    // Encode input text as bert tokens
    let inputTokens = tokenizer.buildModelTokens(sentence: sentence)
    let (inputIds, attentionMask) = tokenizer.buildModelInputs(from: inputTokens)

    print("Sentence: \(sentence)")
    print("Input Tokens: \(inputTokens)")
    print("Input IDs: \(inputIds)")
    print("Attention Mask: \(attentionMask)")
    // Send tokens through the MLModel
    let embeddings = generateMyEmbeddings(inputIds: inputIds, attentionMask: attentionMask)

    return embeddings
}

[LexiestLeszek]

I uploaded the whole thing to google drive along with model

https://drive.google.com/file/d/1x74N1KMmb58scGgJnbi0eMhiOG6hT90L/view?usp=sharing

[ZachNagengast]

Ok I’ll check it out in the AM. What I’m actually looking for the output of would be print(embeddings) in that generateMyEmbeddings function

[LexiestLeszek]

yup, i checked and there are only nans...

Optional([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan])

[ZachNagengast]

Ok well that’s good to know, that probably means the coreml model inference itself is the issue.

[LexiestLeszek]

Any ideas on how to fix it? I dont understand why it works for you and doesnt work for me. ChatGPT didn't gave me any answers too :(

[LexiestLeszek]

@ZachNagengast btw I run it on iPhone 14 simulator, are you running it also on iPhone?

[ZachNagengast]

Ok I'm testing out the app in that google drive and I'm finally able to replicate your issue, investigating...

[LexiestLeszek]

@ZachNagengast I've read that sometimes this issue happens when you use gpu instead of cpu, but for me it doesnt work both ways. My machine is Macbook Pro 2020 M1 by the way.

Did you have any success investigating?

[ZachNagengast]

@LexiestLeszek I'm stumped on this one, at this point it seems like a bug with coremltools right now, I can generate vectors fine through Python but in Swift it's always NaN. Was there a particular reason you wanted to use this model? There are some of other options that you might have better luck with.

[LexiestLeszek]

@ZachNagengast yeah I just wanted a bigger and stronger model. I also tried couple other ones and they returned the same NaNs.

I also converted distilbert (sane that was in the package) by myself and its for some reason 66mb while yours is 85mb. Also doesn't work.

But does it work on your machine or not if you do it? Or is it only my issue?

[LexiestLeszek]

@ZachNagengast I also tried this model https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1

Same result - returns NaNs.

[ZachNagengast]

I had tried that one before too and it actually requires a different tokenizer so the same code won't work by default in Swift until the appropriate tokenizer gets added to this repo. I re-ran the conversion for distilbert and it worked fine for me. I meant to ask, which version of macOS and Xcode is your computer running?

[LexiestLeszek]

@ZachNagengast I have:

• Macbook Pro M1 2020;
• Xcode Version 14.3.1 (14E300c);
• Running all the tests in simulator on iPhone 14.

[LexiestLeszek]

Is there a biggest and strongest possible model that Is applicable to SimilaritySearchKit?

[ZachNagengast]

Just checking hugging face, looks like this one is doing the best right now on information retrieval (and it uses bert tokenizer) https://huggingface.co/thenlper/gte-large. I'll see if this converts any better.

[ZachNagengast]

@LexiestLeszek Ok, good news, this worked 🎉 How I did it: Used my old intel computer with the same pip installation I used to generate the other models. So officially, something is wrong in one or more of the latest python modules on my M2 mac, that is working fine on my old intel mac. torch==2.0.1 coremltools==6.3.0

Here is a google drive of the full project https://drive.google.com/file/d/1e955ELGd1BQSRC05Umtrqz1QGEglXJ5n/view?usp=sharing

The model is significantly slower, but I'd assume the results are much better 👍

[LexiestLeszek]

@ZachNagengast Awesome, thanks a lot! It works!!!!! Such great news!!!

It indeed seems pretty slow, could you please be so kind to also use your magical old intel mac to also convert gte-base and gte-small?

https://huggingface.co/thenlper/gte-base https://huggingface.co/thenlper/gte-small

[ZachNagengast]

Awesome, glad we got to the bottom of it! I just added a new conversion script that should be able to convert the GTE models https://gist.github.com/ZachNagengast/1c1396192bb588801cd0a2f03358c1a7

I narrowed down the issue to the coremltools version 7.0b1, so you should be able to do it yourself if you downgrade to that using pip install coremltools==6.3.0.

[LexiestLeszek]

@ZachNagengast Thanks a lot Zach!!

When I try to convert i see this:

Traceback (most recent call last) ────────────────────────────────╮ │ /Users/leszekmielnikow/Coding/CoreML_usefulstuff/conv_embeddings.py:51 in │ │ │ │ 48 traced_model = torch.jit.trace(wrapped_model.eval(), (encoded_input['input_ids'], encode │ │ 49 traced_model.eval() │ │ 50 │ │ ❱ 51 mlprogram = ct.convert( │ │ 52 │ traced_model, │ │ 53 │ inputs=[ │ │ 54 │ │ ct.TensorType(name="input_ids", shape=(1, 512), dtype=np.int32), │ │ │ │ /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coremltools/conv │ │ erters/_converters_entry.py:492 in convert │ │ │ │ 489 │ if specification_version is None: │ │ 490 │ │ specification_version = _set_default_specification_version(exact_target) │ │ 491 │ │ │ ❱ 492 │ mlmodel = mil_convert( │ │ 493 │ │ model, │ │ 494 │ │ convert_from=exact_source, │ │ 495 │ │ convert_to=exact_target, │ │ │ │ /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coremltools/conv │ │ erters/mil/converter.py:188 in mil_convert │ │ │ │ 185 │ coremltools.converters.mil.Program │ │ 186 │ │ See coremltools.converters.convert │ │ 187 │ """ │ │ ❱ 188 │ return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, com │ │ 189 │ │ 190 │ │ 191 def _mil_convert( │ │ │ │ /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coremltools/conv │ │ erters/mil/converter.py:212 in _mil_convert │ │ │ │ 209 │ │ weights_dir = _tempfile.TemporaryDirectory() │ │ 210 │ │ kwargs["weights_dir"] = weights_dir.name │ │ 211 │ │ │ ❱ 212 │ proto, mil_program = mil_convert_to_proto( │ │ 213 │ │ │ │ │ │ │ model, │ │ 214 │ │ │ │ │ │ │ convert_from, │ │ 215 │ │ │ │ │ │ │ convert_to, │ │ │ │ /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coremltools/conv │ │ erters/mil/converter.py:303 in mil_convert_to_proto │ │ │ │ 300 │ │ │ f"one of: {list(converter_registry.backends.keys())}" │ │ 301 │ │ ) │ │ 302 │ backend_converter = backend_converter_type() │ │ ❱ 303 │ out = backend_converter(prog, kwargs) │ │ 304 │ │ │ 305 │ return out, prog │ │ 306 │ │ │ │ /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coremltools/conv │ │ erters/mil/converter.py:130 in call │ │ │ │ 127 │ def call(self, *args, *kwargs): │ │ 128 │ │ from .backend.mil.load import load as backend_load │ │ 129 │ │ │ │ ❱ 130 │ │ return backend_load(args, kwargs) │ │ 131 │ │ 132 │ │ 133 def _reset_conversion_state(): │ │ │ │ /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coremltools/conv │ │ erters/mil/backend/mil/load.py:283 in load │ │ │ │ 280 │ │ 281 def load(prog, weights_dir, resume_on_errors=False, specification_version=_SPECIFICATION │ │ 282 │ if BlobWriter is None: │ │ ❱ 283 │ │ raise RuntimeError("BlobWriter not loaded") │ │ 284 │ if "main" not in prog.functions: │ │ 285 │ │ raise ValueError("main function not found in program") │ │ 286 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ RuntimeError: BlobWriter not loaded

[ZachNagengast]

@LexiestLeszek I think it wont work with python 3.11, try with 3.10