how to use scraper plugin?

[harrywang]

I tried the scraper plugin and nothing happed - could not find any docs on how to use it.

Then I looked at the code: https://github.com/xlang-ai/OpenAgents/tree/main/real_agents/plugins_agent/plugins/web_scraper

It seems only to only refer to openapi without much information.

Can you give me some pointers how this plugin works?

Thanks.

[Timothyxxx]

Hi, thanks for your question! Here I create a minimal implementation to ping the web_scraper API, it works normally:

Input:

import requests

def call_api(input_json):
    response = requests.post("https://scraper.gafo.tech/scrape", json=input_json)

    if response.status_code == 200:
        return response.json()
    else:
        return {"status_code": response.status_code, "text": response.text}

if __name__ == '__main__':
    print(call_api({"url": "https://www.xlang.ai/", "type": "text"}))

Output:

{'text': 'XLANG Lab\nabout\nteam\nprojects\npublications\nblogs\njoin us\nXLANG Lab\n\nWelcome to the Executable Language Grounding (XLang) Lab! We are part of the HKU NLP Group at the University of Hong Kong. XLang focuses on building language model agents that transform (“grounding”) language instructions into code or actions executable in real-world environments, including databases (data agent), web applications (plugins/web agent), and the physical world (robotic agent) etc,. It lies at the heart of language model agents or natural language interfaces that can interact with and learn from these real-world environments to facilitate human interaction with data analysis, web applications, and robotic instruction through conversation. Recent advances in XLang incorporate techniques such as LLM + external tools, code generation, semantic parsing, efficient and generalizable LLMs, and dialog or interactive systems.\n\nNews\nAug 24, 2023\n🔥🔥 We have released \xa0Lemur70B,\xa0 the SOTA open LLM balancing 📚text & 💻code capabilities! Feel free to download and use the model at \xa0HuggingFace!\xa0\nAug 10, 2023\n🔥🔥 The \xa0XLang Agent Demos\xa0 are now available. Feel free to check! For more details about the XLang framework, please stay tuned at our \xa0GitHub repo\xa0 and follow our \xa0twitter\nAug 8, 2023\nThe group website is now live!\nXLANG Agents\nOpen-source framework and ecosystem for building and evaluating LLM-based agents\nTry Online Demo\n\nOur ongoing effort to build an open-source framework and ecosystem for building and evaluating language model agents. The open-source journey begins with XLang Agent demos. In the following months, and beyond, we will be open-sourcing several significant projects, including a framework, models, methods, benchmarks, and more. In the foreseeable future, we envision that a proficient functional agent will require the fusion of these various agents.\n\nData Agent\nproficient in data-centric tasks with code execution and efficient data search, manipulation and visualization tools.\nPlugins Agent\ncatering to various daily life tasks with over 200 third-party plugins for enhanced assistance\nWeb Agent\nstreamlining web browsing, making information retrieval and resource access easier\nRobotic Agent\ntranslating language instructions into robotic actions to carry out tasks in the physical real world.\nAcknowledgements\n\nWe thank the following institutions for their gift support: Google Research, Amazon AWS and Salesforce Research.\n\nDonations appreciated! XLANG Lab is funded by university grants and kind donations. We welcome any form of donations and sponsorships. To aid our mission of building open-source language agents, contact us at mail@xlang.ai.\n\n© Copyright 2023 XLANG Lab. All right reserved.'}

Then allow me to explain the logic of plugin~ Actually, it is (almost) the same logic with OpenAI ChatGPT plugins, inside each plugin file, let's take Klarna (https://github.com/xlang-ai/OpenAgents/tree/main/real_agents/plugins_agent/plugins/klarna) for example:

It contains three parts: paths, ai-plugin.json, and openapi.yaml

The ai-plugin.json provides meta information of the plugin: https://github.com/xlang-ai/OpenAgents/blob/main/real_agents/plugins_agent/plugins/klarna/ai-plugin.json

the openapi.yaml describes the API information for calling, if you are unfamiliar with it, check this https://swagger.io/specification/

and insides the paths, we implement the paths (API functions) of the API endpoint, for example, the /public/openai/v0/products of klarna

We unify every API call to the same function call_api with different parameters, for LLM to better generate, in future, this logic can be refactored with other powerful foudations!!

[harrywang]

Thanks!