Failed to scrape folder for docs

[wizche]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Debian 12
- Continue: v0.9.159
- IDE: VS Code

Description

When serving a website via python3 -m http.server scraper doesnt construct the URLs correctly.

To reproduce

1. Create a structure similar to the one in the screenshot
2. Serve the website
3. Start the scraper via Add docs...
4. page2.html is called on root (/) instead of /sub path

Log output

[Extension Host] Starting crawl from:  http://localhost:8000/  - Max Depth:  3
[Extension Host] Crawl completed
[Extension Host] Creating Embeddings for  1  articles
[Extension Host] Adding  0  embeddings to db
[Extension Host] Error handling webview message: {
  "msg": {
    "messageId": "f40d8f31-2ae1-4ead-85e6-27fc93ffc0e8",
    "messageType": "context/addDocs",
    "data": {
      "startUrl": "http://localhost:8000/",
      "rootUrl": "http://localhost:8000/",
      "title": "test",
      "maxDepth": 3
    }
  }
}

Error: Either data or schema needs to defined

[sestinj]

@wizche Any chance you could .zip the contents of this example site so I can test the literal exact same thing? I imagine this would make it quite straightforward to solve the problem

[wizche]

Hi Nate, Its literally just a folder with an empty file on it. My guess is that the scraper always joins path with the base URL and not with the actual parent

[cmann50]

Let me know if this requires a separate ticket. I am using the JetBrains plugin on OSX and trying to add documentation using @docs. It lets me specify the URL to the website with the documentation and a name. I hit “OK”, and after a few seconds, a floating popup appears along with this error in the logs: “Either data or schema needs to be defined.”

There are no other errors in the logs.

[info] Starting Continue core...
[2024-06-24T14:27:15] [info] Starting Continue core... 
[2024-06-24T14:28:04] Error running handler for "context/addDocs":  Error: Either data or schema needs to defined
[2024-06-24T14:28:04] Error: Either data or schema needs to defined 

I am using version 0.0.47 of the plugin from the EAP url listed in the troubleshooting page. But I believe the version I was on before using the EAP url was higher, but also didn't work.

[edneyreis999]

@cmann50 I had exactly the same problem. I'll post my solution here. If there is a better place, please move the answer to another topic.

The extension saves the documents in an SQLite database. .continue/index/docs.sqlite You may still need to create the schema. Please just go ahead and index any docs that Continue already provides before creating your own. Or create the schema manually.

I hope it solves your problem.

[cmann50]

Thanks, I tried reinstalling and now it says the @docs context provide is not supported in Jetbrains IDE, we'll have an update soon.