search

zotero / scaffold

Zotero translator creation IDE
http://www.zotero.org/support/dev/scaffold
38 stars 9 forks source link

Error: No title specified for item #38

Open owcz opened 7 years ago

owcz commented 7 years ago

I have two translators that work fine in Scaffold (Blouin Artinfo and Destructoid) with doWeb and in deployment, but for some reason, they throw a Error: No title specified for item in their Scaffold test whenever FW.Xpath is called in the scraper's title field. I get the following errors no matter what XPath, even if, for instance, I duplicate the XPath that correctly works for the date. Again, the error only occurs in Scaffold's test conditions and not when using doWeb or in practice in the browser.

It's possible to work around the error by setting the title to use a dummy string instead of the XPath. The URL will pass the test this way (and I can manually correct the rest of the details), but oddly, the test will not include the creators, date, or abstractNote (the other fields that also use XPath). To reiterate, this only happens to specific scrapers, and only in the Scaffold test environment (doWeb and browser extension continue to work fine with the translator). To work around this, I set the line to use a dummy string instead of the XPath just to pass the test, and I manually edit the correct title into the output and just avoid running tests again, but of course this isn't a permanent solution.

In Blouin Artinfo, the scraper's XPath is on line 31. (Do also mind that the translator has multiple scrapers and this only occurs in the one containing line 31.) If not set to a dummy string, this is what Scaffold spits out on test:

16:49:42 TranslatorTester: Running 1 test for Blouin Artinfo
16:49:42 TranslatorTester: Running Blouin Artinfo Test 1
16:49:42 TranslatorTester: Translating http://www.blouinartinfo.com/news/story/2003327/to-view-a-world-within-a-grain-of-sand-at-sakura-city-museum
16:49:42 Translation using Blouin Artinfo failed: 
         string => Error: No title specified for item
         stack => Zotero.Translate.Sandbox.Web._itemDone@chrome://zotero/content/xpcom/translation/translate.js:593:32
         Zotero.Translate.SandboxManager.prototype.importObject/attachTo[localKey]<@chrome://zotero/content/xpcom/translation/translate_firefox.js:503:29
         @resource://gre/modules/RemoteAddonsParent.jsm:780:66
         Zotero.Item.prototype.complete@resource://gre/modules/RemoteAddonsParent.jsm:780:277
         FW.doWeb/<@Blouin Artinfo:2:8445
         FW._Scraper/this.makeItems@Blouin Artinfo:2:3003
         FW.doWeb@Blouin Artinfo:2:8354
         doWeb@Blouin Artinfo:28:35
         Zotero.Translate.Base.prototype._translateTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1221:4
         Zotero.Translate.Web.prototype._translateTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1814:3
         Zotero.Translate.Base.prototype.translate/<@chrome://zotero/content/xpcom/translation/translate.js:1188:58
         Zotero.Translate.Base.prototype._loadTranslator@chrome://zotero/content/xpcom/translation/translate.js:1573:16
         Zotero.Translate.Base.prototype.translate@chrome://zotero/content/xpcom/translation/translate.js:1188:4
         Zotero.Translate.Web.prototype.translate@chrome://zotero/content/xpcom/translation/translate.js:1805:2
         Zotero_TranslatorTester.prototype._runTestTranslate@chrome://zotero/content/tools/testTranslators/translatorTester.js:533:2
         Zotero_TranslatorTester.prototype.runTest/<@chrome://zotero/content/tools/testTranslators/translatorTester.js:464:3
         Zotero.Translate.Base.prototype._runHandler@chrome://zotero/content/xpcom/translation/translate.js:1021:20
         Zotero.Translate.Base.prototype._detectTranslatorsCollected@chrome://zotero/content/xpcom/translation/translate.js:1538:3
         Zotero.Translate.Base.prototype.complete@chrome://zotero/content/xpcom/translation/translate.js:1373:30
         Zotero.Translate.Web.prototype.complete@chrome://zotero/content/xpcom/translation/translate.js:1936:20
         Zotero.Translate.Base.prototype.decrementAsyncProcesses@chrome://zotero/content/xpcom/translation/translate.js:992:4
         Zotero.Translate.Base.prototype._detectTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1522:3
         Zotero.Translate.Base.prototype._detect/<@chrome://zotero/content/xpcom/translation/translate.js:1503:17
         Zotero.Translate.Base.prototype._loadTranslator@chrome://zotero/content/xpcom/translation/translate.js:1573:16
         Zotero.Translate.Base.prototype._detect@chrome://zotero/content/xpcom/translation/translate.js:1502:3
         Zotero_TranslatorTester.prototype.runTest@chrome://zotero/content/tools/testTranslators/translatorTester.js:514:2
         Zotero_TranslatorTester.prototype.fetchPageAndRunTest/runTest@chrome://zotero/content/tools/testTranslators/translatorTester.js:410:3
         Zotero_TranslatorTester.prototype.fetchPageAndRunTest/hiddenBrowser<@chrome://zotero/content/tools/testTranslators/translatorTester.js:427:5
         Zotero.HTTP</this.processDocuments/onLoad@chrome://zotero/content/xpcom/http.js:881:5

         url => http://www.blouinartinfo.com/news/story/2003327/to-view-a-world-within-a-grain-of-sand-at-sakura-city-museum
         downloadAssociatedFiles => true
         automaticSnapshots => true
16:49:42 TranslatorTester: Blouin Artinfo Test 1: failed (Translation failed: Error: No title specified for item)

In Destructoid, the title is also on line 31.

16:59:15 TranslatorTester: Running 1 test for Destructoid
16:59:15 TranslatorTester: Running Destructoid Test 1
16:59:15 TranslatorTester: Translating https://www.destructoid.com/review-videoball-374456.phtml
16:59:15 Translation using Destructoid failed: 
         string => Error: No title specified for item
         stack => Zotero.Translate.Sandbox.Web._itemDone@chrome://zotero/content/xpcom/translation/translate.js:593:32
         Zotero.Translate.SandboxManager.prototype.importObject/attachTo[localKey]<@chrome://zotero/content/xpcom/translation/translate_firefox.js:503:29
         @resource://gre/modules/RemoteAddonsParent.jsm:780:66
         Zotero.Item.prototype.complete@resource://gre/modules/RemoteAddonsParent.jsm:780:277
         FW.doWeb/<@Destructoid:2:8445
         FW._Scraper/this.makeItems@Destructoid:2:3003
         FW.doWeb@Destructoid:2:8354
         doWeb@Destructoid:28:35
         Zotero.Translate.Base.prototype._translateTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1221:4
         Zotero.Translate.Web.prototype._translateTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1814:3
         Zotero.Translate.Base.prototype.translate/<@chrome://zotero/content/xpcom/translation/translate.js:1188:58
         Zotero.Translate.Base.prototype._loadTranslator@chrome://zotero/content/xpcom/translation/translate.js:1573:16
         Zotero.Translate.Base.prototype.translate@chrome://zotero/content/xpcom/translation/translate.js:1188:4
         Zotero.Translate.Web.prototype.translate@chrome://zotero/content/xpcom/translation/translate.js:1805:2
         Zotero_TranslatorTester.prototype._runTestTranslate@chrome://zotero/content/tools/testTranslators/translatorTester.js:533:2
         Zotero_TranslatorTester.prototype.runTest/<@chrome://zotero/content/tools/testTranslators/translatorTester.js:464:3
         Zotero.Translate.Base.prototype._runHandler@chrome://zotero/content/xpcom/translation/translate.js:1021:20
         Zotero.Translate.Base.prototype._detectTranslatorsCollected@chrome://zotero/content/xpcom/translation/translate.js:1538:3
         Zotero.Translate.Base.prototype.complete@chrome://zotero/content/xpcom/translation/translate.js:1373:30
         Zotero.Translate.Web.prototype.complete@chrome://zotero/content/xpcom/translation/translate.js:1936:20
         Zotero.Translate.Base.prototype.decrementAsyncProcesses@chrome://zotero/content/xpcom/translation/translate.js:992:4
         Zotero.Translate.Base.prototype._detectTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1522:3
         Zotero.Translate.Base.prototype._detect/<@chrome://zotero/content/xpcom/translation/translate.js:1503:17
         Zotero.Translate.Base.prototype._loadTranslator@chrome://zotero/content/xpcom/translation/translate.js:1573:16
         Zotero.Translate.Base.prototype._detect@chrome://zotero/content/xpcom/translation/translate.js:1502:3
         Zotero_TranslatorTester.prototype.runTest@chrome://zotero/content/tools/testTranslators/translatorTester.js:514:2
         Zotero_TranslatorTester.prototype.fetchPageAndRunTest/runTest@chrome://zotero/content/tools/testTranslators/translatorTester.js:410:3
         Zotero_TranslatorTester.prototype.fetchPageAndRunTest/hiddenBrowser<@chrome://zotero/content/tools/testTranslators/translatorTester.js:427:5
         Zotero.HTTP</this.processDocuments/onLoad@chrome://zotero/content/xpcom/http.js:881:5

         url => https://www.destructoid.com/review-videoball-374456.phtml
         downloadAssociatedFiles => true
         automaticSnapshots => true
16:59:15 TranslatorTester: Destructoid Test 1: failed (Translation failed: Error: No title specified for item)

Should be up to date on all updates, if that helps

owcz commented 7 years ago

Also when testing https://theintercept.com/document/2017/02/22/mastering-the-internet/ in zotero/translators/pull/1365 (runs fine with doWeb, only issue is in Scaffold test)

16:00:30 TranslatorTester: Running 1 test for The Intercept
16:00:30 TranslatorTester: Running The Intercept Test 1
16:00:31 TranslatorTester: Translating https://theintercept.com/document/2017/02/22/mastering-the-internet/
16:00:31 Translation using The Intercept failed: 
         string => Error: No title specified for item
         stack => _itemDone@chrome://zotero/content/xpcom/translation/translate.js:593:32
         importObject/attachTo[localKey]<@chrome://zotero/content/xpcom/translation/translate_firefox.js:503:29
         @resource://gre/modules/RemoteAddonsParent.jsm:813:66
         Zotero.Item.prototype.complete@resource://gre/modules/RemoteAddonsParent.jsm:813:277
         scrapeDocument@The Intercept:85:2
         doWeb@The Intercept:117:3
         _translateTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1221:4
         Zotero.Translate.Web.prototype._translateTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1814:3
         translate/<@chrome://zotero/content/xpcom/translation/translate.js:1188:58
         _loadTranslator@chrome://zotero/content/xpcom/translation/translate.js:1573:16
         translate@chrome://zotero/content/xpcom/translation/translate.js:1188:4
         Zotero.Translate.Web.prototype.translate@chrome://zotero/content/xpcom/translation/translate.js:1805:2
         Zotero_TranslatorTester.prototype._runTestTranslate@chrome://zotero/content/tools/testTranslators/translatorTester.js:533:2
         Zotero_TranslatorTester.prototype.runTest/<@chrome://zotero/content/tools/testTranslators/translatorTester.js:464:3
         _runHandler@chrome://zotero/content/xpcom/translation/translate.js:1021:20
         _detectTranslatorsCollected@chrome://zotero/content/xpcom/translation/translate.js:1538:3
         complete@chrome://zotero/content/xpcom/translation/translate.js:1373:30
         Zotero.Translate.Web.prototype.complete@chrome://zotero/content/xpcom/translation/translate.js:1936:20
         decrementAsyncProcesses@chrome://zotero/content/xpcom/translation/translate.js:992:4
         _detectTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1522:3
         _detect/<@chrome://zotero/content/xpcom/translation/translate.js:1503:17
         _loadTranslator@chrome://zotero/content/xpcom/translation/translate.js:1573:16
         _detect@chrome://zotero/content/xpcom/translation/translate.js:1502:3
         Zotero_TranslatorTester.prototype.runTest@chrome://zotero/content/tools/testTranslators/translatorTester.js:514:2
         runTest@chrome://zotero/content/tools/testTranslators/translatorTester.js:410:3
         Zotero_TranslatorTester.prototype.fetchPageAndRunTest/hiddenBrowser<@chrome://zotero/content/tools/testTranslators/translatorTester.js:427:5
         onLoad@chrome://zotero/content/xpcom/http.js:881:5

         url => https://theintercept.com/document/2017/02/22/mastering-the-internet/
         downloadAssociatedFiles => true
         automaticSnapshots => true
16:00:31 TranslatorTester: The Intercept Test 1: failed (Translation failed: Error: No title specified for item)
zuphilip commented 7 years ago

I looked at The Intercept and it looks that the content is loaded dynamically. Thus, I guess that Zotero tries to scrape the page, when it is not fully loaded. It works in the browser because there the page can be loaded completely. But it does not work in Scaffold or server (e.g. Citoid). Maybe the other two examples are similar...

adam3smith commented 7 years ago

does the server understand monitorDOMChanges? No, right?

zuphilip commented 7 years ago

I think 'monitorDOMChangeworks only indetectWeb`.

owcz commented 6 years ago

circling back to this: rewrote Destructoid translator without FW and still hitting this issue

first, there are no issues with running the translator, only when doing test conditions in Scaffold

if I use EM for most of the translator but just add item.title = "dummy"; in the translator.setHandler... section, both the translator and the test will run fine. what's curious, though, is that the test will mark the translation as successful even when it hasn't pulled from EM at all. so I can run "update" on a URL, it will be added to the "Tests" section while missing all the EM metadata, and if I go to "Run" test on the same URL, the translator correctly notes the discrepancy between the saved test and what the results should be.