Now that XHR processDocuments() has landed, we should start looking out for translators that need to be updated to work with the new method. This is a tracking bug for those issues, which can reference this issue.
To review, any content that's generated client-side will be unavailable in the document object provided by processDocuments(), which will now show only the initial page structure that came over the wire. However, the fact that client-side JavaScript can generate the page means that we can almost certainly do so as well, and possibly in a more reliable, structured way than we were doing before with page scraping (and definitely much more quickly than waiting for the page to load in a hidden browser). The best way to debug these sorts of pages is to use the Network tab of the browser devtools and monitor API requests that the page is making. In most cases we should be able to duplicate those requests (which hopefully are cached, avoiding additional network requests) and use the same data the page is using. In some cases, pages will instead embed JSON in the page itself, which we can also extract.
Note that the signature of processDocuments() has changed (though the old signature will still work for existing translators). Instead of done and exception callbacks, it now returns a promise. The processor callback can now return a promise as well, which will be waited for before moving onto the next URL. Instead of using doGet()/doPost()/etc., translators should use request(), which returns a promise and properly propagates errors. While I haven't yet tested this, it should now be possible to use async/await in translators, which can make working with promises much easier. If anyone knows of a translator affected by this change, I'm happy to work up an example update to serve as a template (and to verify that everything is actually working as expected).
Also note that this only affects processDocuments(), so current-page saves still have access to generated content. Whether it makes sense to use that directly when available or to keep the multiple code consistent might vary by site (perhaps by whether the API requests are actually cached).
The new processDocuments() is available now in the 5.0 Beta, so it can be tested with Scaffold. It hasn't been rolled out to the connector yet, but it should work the same there. translation-server has always used XHR for this, so any multiple results that fail there might point to a problem (though I'm seeing some timeouts in the current test results that might point to a separate problem).
Version 5.0.25 of the connector is now out with this change, so we should keep an eye out for multiple breakage on sites that generate content client-side.
Now that XHR processDocuments() has landed, we should start looking out for translators that need to be updated to work with the new method. This is a tracking bug for those issues, which can reference this issue.
To review, any content that's generated client-side will be unavailable in the
document
object provided byprocessDocuments()
, which will now show only the initial page structure that came over the wire. However, the fact that client-side JavaScript can generate the page means that we can almost certainly do so as well, and possibly in a more reliable, structured way than we were doing before with page scraping (and definitely much more quickly than waiting for the page to load in a hidden browser). The best way to debug these sorts of pages is to use the Network tab of the browser devtools and monitor API requests that the page is making. In most cases we should be able to duplicate those requests (which hopefully are cached, avoiding additional network requests) and use the same data the page is using. In some cases, pages will instead embed JSON in the page itself, which we can also extract.Note that the signature of
processDocuments()
has changed (though the old signature will still work for existing translators). Instead ofdone
andexception
callbacks, it now returns a promise. Theprocessor
callback can now return a promise as well, which will be waited for before moving onto the next URL. Instead of usingdoGet()
/doPost()
/etc., translators should userequest()
, which returns a promise and properly propagates errors. While I haven't yet tested this, it should now be possible to useasync
/await
in translators, which can make working with promises much easier. If anyone knows of a translator affected by this change, I'm happy to work up an example update to serve as a template (and to verify that everything is actually working as expected).Also note that this only affects
processDocuments()
, so current-page saves still have access to generated content. Whether it makes sense to use that directly when available or to keep themultiple
code consistent might vary by site (perhaps by whether the API requests are actually cached).The new
processDocuments()
is available now in the 5.0 Beta, so it can be tested with Scaffold. It hasn't been rolled out to the connector yet, but it should work the same there. translation-server has always used XHR for this, so anymultiple
results that fail there might point to a problem (though I'm seeing some timeouts in the current test results that might point to a separate problem).