Could you also consider adding example on how to add custom API's? I think this approach might be generalizable to other kind of API's, such as search API's.

[sonichi]

          Could you also consider adding example on how to add custom API's? I think this approach might be generalizable to other kind of API's, such as search API's.

Originally posted by @torronen in https://github.com/microsoft/FLAML/issues/954#issuecomment-1528163090

[sonichi]

@torronen I'd like to understand more about what you mean by "add custom API's". Could you elaborate?

[torronen]

I believe this should already be doable with FLAML, so maybe just needs a sample.

General examples: 1.users could add themselves other engines, such as gpt-3.5-turbo-0301 or their own finetune OpenAI model names 2.users could use other LLM providers than OpenAI 3.users could "finetune" any kind of endpoints that provide any kind of parameters

I may think about this in a too complex way, but I am thinking something like this

 default_search_space = {
    "url": tune.choice([
        "https://api.example.org",
         "https://api-alternative.example.org"
    ]),
    "verb": "POST",
    "json_structure": "temperature: {temperature}, top_p: {top_p}",   // optional, if not specified, then flat JSON
    "temperature_or_top_p": tune.choice(
        [
            {"temperature": tune.uniform(0, 1)},
            {"top_p": tune.uniform(0, 1)},
        ]
    ),
    "max_tokens": tune.lograndint(50, 1000),
    "n": tune.randint(1, 100),
    "prompt": "{prompt}",
}

My specific case: I am working on degree and occupation suggestion problem. Users (students, people looking for a new career) etc. will provide a list of data (personality test results, work history, general interests, preferences, goals etc.) and my task is to provide them potential occupation from ESCO classification. In the next step, I will need to find suitable degrees for the selected occupation. For example, user may select "software developer" so the app should suggest a list of university, college and vocational degrees, potentially ranking them. It is more challenging for other occupation, such as primary school teacher, where there is only 1 or few correct answers but there are many other degrees with similar or related content.

The first version is complete. The endpoint uses mixture of models to give the predictions. For each model and data preprocessing there are a few parameters. There are also other parameters for the matching service, such as spellchecking tolerance, weight given for each data point and so on. I have also a few options for the search strategy, such as rank everything, or first drop 90% then rank etc.

When reading about the OpenAI integration, I thought I might be able to use this to get "finetune" this service. It is not a model but I could expose all parameters and then finetune it like it were a model. Perhaps it could even one way to build different user profiles in the future.

[sonichi]

I believe this should already be doable with FLAML, so maybe just needs a sample.

General examples: 1.users could add themselves other engines, such as gpt-3.5-turbo-0301 or their own finetune OpenAI model names 2.users could use other LLM providers than OpenAI 3.users could "finetune" any kind of endpoints that provide any kind of parameters

I may think about this in a too complex way, but I am thinking something like this

 default_search_space = {
    "url": tune.choice([
        "https://api.example.org",
         "https://api-alternative.example.org"
    ]),
    "verb": "POST",
    "json_structure": "temperature: {temperature}, top_p: {top_p}",   // optional, if not specified, then flat JSON
    "temperature_or_top_p": tune.choice(
        [
            {"temperature": tune.uniform(0, 1)},
            {"top_p": tune.uniform(0, 1)},
        ]
    ),
    "max_tokens": tune.lograndint(50, 1000),
    "n": tune.randint(1, 100),
    "prompt": "{prompt}",
}

My specific case: I am working on degree and occupation suggestion problem. Users (students, people looking for a new career) etc. will provide a list of data (personality test results, work history, general interests, preferences, goals etc.) and my task is to provide them potential occupation from ESCO classification. In the next step, I will need to find suitable degrees for the selected occupation. For example, user may select "software developer" so the app should suggest a list of university, college and vocational degrees, potentially ranking them. It is more challenging for other occupation, such as primary school teacher, where there is only 1 or few correct answers but there are many other degrees with similar or related content.

The first version is complete. The endpoint uses mixture of models to give the predictions. For each model and data preprocessing there are a few parameters. There are also other parameters for the matching service, such as spellchecking tolerance, weight given for each data point and so on. I have also a few options for the search strategy, such as rank everything, or first drop 90% then rank etc.

When reading about the OpenAI integration, I thought I might be able to use this to get "finetune" this service. It is not a model but I could expose all parameters and then finetune it like it were a model. Perhaps it could even one way to build different user profiles in the future.

Thanks for the elaboration. (1) is easy to do with the current flaml.oai.tune by specifying model. (2) and (3) are feasible with flaml.tune but not with flaml.oai.tune currently. Are you familiar with how flaml.tune works?

[torronen]

@sonichi not yet, but the docs seem clear enough as a starting point. I think I tried it a bit a long time ago, but I am not sure. Thanks for the tip!

(1) Can we put any model name as parameter and it should work, including finetuned ones?

BTW, I am not sure if it is interesting or not, but I learned about a project that aims to re-create OpenAI API for local models, such as LLaMa. Only thing that needs updating is the endpoint address. I have not yet tried it and seems I now misremember the name.

[sonichi]

@sonichi not yet, but the docs seem clear enough as a starting point. I think I tried it a bit a long time ago, but I am not sure. Thanks for the tip!

(1) Can we put any model name as parameter and it should work, including finetuned ones?

According to OpenAI's documentation, (1) should work for fine-tuned models. I have only tested non-fine-tuned models. If you get a chance to test fine-tuned model, I'll be curious to know.

BTW, I am not sure if it is interesting or not, but I learned about a project that aims to re-create OpenAI API for local models, such as LLaMa. Only thing that needs updating is the endpoint address. I have not yet tried it and seems I now misremember the name.

It sounds interesting to me. That'll make things a lot easier.

[sonichi]

@torronen Is this the project you mentioned? https://github.com/go-skynet/LocalAI

It's written in Go.

[sonichi]

@torronen Is this the project you mentioned? https://github.com/go-skynet/LocalAI

It's written in Go.

A python project is https://github.com/lhenault/simpleAI

[torronen]

@sonichi Yes, that is the one. I just remembered it was something to do with word "village" 🥇 Close enough to localAI 😆

[sonichi]

@sonichi Yes, that is the one. I just remembered it was something to do with word "village" 🥇 Close enough to localAI 😆

I added some documentation about it: https://microsoft.github.io/FLAML/docs/Use-Cases/Auto-Generation/#api-unification and added a feature to try multiple endpoints in turn: https://microsoft.github.io/FLAML/docs/Use-Cases/Auto-Generation/#error-handling Let me know if it works!

[torronen]

@sonichi Amazing, thanks!