Building rw-agent for use by RedwoodJS developers

[adriatic]

aa

[adriatic]

I got the corpusID 1b40e16f-3783-4c1a-b23e-b598ee1771be from Ben to help me accelerate my project (Building rw-agent for use by RedwoodJS developers). I started the recrawl task to catch all updates to rw help created in last two months.

After starting this task the UI remained mostly unchanged (only the Ready flag changed to Crawling) and there was no indication of any progress for the longest time. I watched the Chrome console and got this

GET https://www.googletagmanager.com/gtag/js?id=G-T4VHHYL14W net::ERR_BLOCKED_BY_CLIENT
1b40e16f-3783-4c1a-b23e-b598ee1771be:72 

GET https://cdn.heapanalytics.com/js/heap-2470912105.js net::ERR_BLOCKED_BY_CLIENT
heap.load @ 1b40e16f-3783-4c1a-b23e-b598ee1771be:72
(anonymous) @ 1b40e16f-3783-4c1a-b23e-b598ee1771be:73
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] LaunchDarkly client initialized
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] Opening stream connection to https://clientstream.launchdarkly.com/eval/656a0dea675b570f2e9f2dff/eyJhbm9ueW1vdXMiOnRydWUsImtpbmQiOiJ1c2VyIiwia2V5IjoiNTMwN2VmMTAtYTFkYy0xMWVlLWE0OTItNTcxNjU3NjBhNzZhIn0
index-bundle.1f933f9dd88b.js:2 Identifying user with LD
index-bundle.1f933f9dd88b.js:2 {userId: '2534', email: 'myrw.agent@gmail.com', avatarUrl: 'https://www.gravatar.com/avatar/283c15f81990efb323ac5f87ed5e6fd0?d=mp&s=40', created: '2024-01-16T22:13:44.409838Z', modified: '2024-01-16T22:13:55.538419Z', …}
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] Closing stream connection
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] Opening stream connection to https://clientstream.launchdarkly.com/eval/656a0dea675b570f2e9f2dff/eyJraW5kIjoidXNlciIsImtleSI6InVzZXI6MjUzNCIsIm5hbWUiOiIiLCJlbWFpbCI6Im15cncuYWdlbnRAZ21haWwuY29tIiwiYW5vbnltb3VzIjpmYWxzZX0
271-7d61ba805cf66fa8.js:1 Datadog logging initialized {}
271-7d61ba805cf66fa8.js:1 Datadog logging initialized {}
embed-launcher:6 [embed:host protocol][/embed-launcher] embed got postMessage from host MessageEvent {isTrusted: true, data: 'channel-message-port', origin: 'https://console.fixie.ai', lastEventId: '', source: Window, …}
8The resource <URL> was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally.
index-bundle.1f933f9dd88b.js:2 [batch] MIXPANEL REQUEST: [{…}]
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] Error on stream connection: {"isTrusted":true}, will continue retrying after 603 milliseconds.
overrideMethod @ console.js:213
(anonymous) @ index-bundle.1f933f9dd88b.js:2
s @ index-bundle.1f933f9dd88b.js:2
l.<computed> @ index-bundle.1f933f9dd88b.js:2
b @ index-bundle.1f933f9dd88b.js:2
Show 1 more frame
Show less
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] Closing stream connection
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] Opening stream connection to https://clientstream.launchdarkly.com/eval/656a0dea675b570f2e9f2dff/eyJraW5kIjoidXNlciIsImtleSI6InVzZXI6MjUzNCIsIm5hbWUiOiIiLCJlbWFpbCI6Im15cncuYWdlbnRAZ21haWwuY29tIiwiYW5vbnltb3VzIjpmYWxzZX0
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] Error on stream connection: {"isTrusted":true}, will continue retrying after 647 milliseconds.
overrideMethod @ console.js:213
(anonymous) @ index-bundle.1f933f9dd88b.js:2
s @ index-bundle.1f933f9dd88b.js:2
l.<computed> @ index-bundle.1f933f9dd88b.js:2
b @ index-bundle.1f933f9dd88b.js:2
error (async)
x @ index-bundle.1f933f9dd88b.js:2
setTimeout (async)
w @ index-bundle.1f933f9dd88b.js:2
b @ index-bundle.1f933f9dd88b.js:2
Show 1 more frame
Show less
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] Closing stream connection
index-bundle.1f933f9dd88b.js:2 [LaunchDarkly] Opening stream connection to https://clientstream.launchdarkly.com/eval/656a0dea675b570f2e9f2dff/eyJraW5kIjoidXNlciIsImtleSI6InVzZXI6MjUzNCIsIm5hbWUiOiIiLCJlbWFpbCI6Im15cncuYWdlbnRAZ21haWwuY29tIiwiYW5vbnltb3VzIjpmYWxzZX0

There are several error conditions in this log, and it seems that the task failed (based on number of chunks and vectors)

[benlower]

Per our chat on Discord, sharing here in case other will see this. This is because the start URL of redwoodjs.com/docs actually immediately redirects to .../docs/index but all the content is hung below /docs. The fix for this is to set include pattern using the API.