Hi, I am going to share some data with you guys.

[ValorLin]

Hi, I am interested in this project and I have done some reasearch. I tried the evaluation on the TorchHub API senerio with some of the latest models and other fine-tuned models.

• The gorilla-ada/babbage/curie/davinci are fine-tuned models base on OpenAI's ada/babbage/curie/davinci models. Trained with the data in this project (torchhub_train.json) and got suprising result. They are much better, the gorilla-babbage model gets a 90.86% accuracy.
• The latest gpt-3.5-turbo get 29.57% improvement compared to gpt-3.5-turbo-0301 and gpt-4 get 22.05% improvment compared to gpt-4-0315. Here is the result:

Below is a example of the training data that I used for fine-tuning on OpenAI models.

{"prompt":"I need an API that can estimate the relative depth of objects from a single image. The output should be an inverse depth map.\n\n###\n\n","completion":" {'domain': 'Classification', 'api_call': \"model = torch.hub.load('intel-isl\/MiDaS', 'DPT_Hybrid', pretrained=True)\", 'api_provider': 'PyTorch', 'explanation': \"Load the pretrained MiDaS model for estimating relative depth from a single image, which outputs an inverse depth map.\", 'code': 'import torch\nmodel = torch.hub.load(\"intel-isl\/MiDaS\", \"DPT_Hybrid\", pretrained=True)'}###"}

The full converted training data can be found here: https://gist.github.com/ValorLin/807ac5f75361775dfd2cc31f21a03e77

I hope that data is helpful. Its very interesting result, you should try it.

[ShishirPatil]

Hey @ValorLin thank's for sharing this! Super insightful. Yeah, one thing we might want to keep in mind is that OpenAI's models are slightly larger, vs the 7B Gorilla models - so that might explain the performance (?) Or potentially better pre-training data? Would be interesting for the open source community to explore.

BTW do you mind writing a blog or a write-up, with your pipeline including the data-generation, results, insights on where it does well, not so well, etc ! Would be a great read for the community - including and especially me 😄

Nice job!