search

nextcloud / news-updater

:newspaper: Fast, parallel feed updater for the News app; written in Python
GNU General Public License v3.0
107 stars 23 forks source link

News 15.1 and updater 11 with REST #29

Closed freaktechnik closed 3 years ago

freaktechnik commented 3 years ago

I've updated nextcloud news and the updater. I've since only gotten errors, trying all three API versions.

v1-2: I think this is what should work, and according to info logs it does for the first few feeds, but after the first few request all of them get the same error:

2020-12-09 17:00:01,854 - Nextcloud News Updater - INFO - Running update once with 10 threads
2020-12-09 17:00:01,854 - Nextcloud News Updater - INFO - Calling before update url:  http://localhost:8889/index.php/apps/news/api/v1-2/cleanup/before-update
2020-12-09 17:00:02,371 - Nextcloud News Updater - INFO - Received these feeds to update: {"feeds":[{"id":1,"userId":"normal"},{"id":2,"userId":"normal"},{"id":3,"userId":"normal"},{"id":4,"userId":"normal"},{"id":5,"userId":"normal"},{"id":6,"userId":"normal"},{"id":7,"userId":"normal"},{"id":8,"userId":"normal"},{"id":9,"userId":"normal"},{"id":10,"userId":"normal"},{"id":11,"userId":"normal"},{"id":12,"userId":"normal"},{"id":13,"userId":"normal"},{"id":14,"userId":"normal"},{"id":15,"userId":"normal"},{"id":16,"userId":"normal"},{"id":17,"userId":"normal"},{"id":18,"userId":"normal"},{"id":19,"userId":"normal"},{"id":20,"userId":"normal"},{"id":21,"userId":"normal"},{"id":22,"userId":"normal"},{"id":23,"userId":"normal"},{"id":24,"userId":"normal"},{"id":25,"userId":"normal"},{"id":26,"userId":"normal"},{"id":27,"userId":"normal"},{"id":28,"userId":"normal"},{"id":29,"userId":"normal"},{"id":30,"userId":"normal"},{"id":31,"userId":"normal"},{"id":32,"userId":"normal"},{"id":33,"u
 serId":"normal"},{"id":34,"userId":"normal"},{"id":35,"userId":"normal"},{"id":36,"userId":"normal"},{"id":37,"userId":"normal"},{"id":38,"userId":"normal"},{"id":39,"userId":"normal"},{"id":40,"userId":"normal"},{"id":41,"userId":"normal"},{"id":42,"userId":"normal"},{"id":43,"userId":"normal"},{"id":44,"userId":"normal"},{"id":45,"userId":"normal"},{"id":46,"userId":"normal"},{"id":47,"userId":"normal"},{"id":48,"userId":"normal"},{"id":49,"userId":"normal"},{"id":50,"userId":"normal"},{"id":51,"userId":"normal"},{"id":52,"userId":"normal"},{"id":53,"userId":"normal"},{"id":54,"userId":"normal"},{"id":56,"userId":"normal"},{"id":57,"userId":"normal"},{"id":58,"userId":"normal"},{"id":59,"userId":"normal"},{"id":60,"userId":"normal"},{"id":61,"userId":"normal"},{"id":62,"userId":"normal"},{"id":63,"userId":"normal"},{"id":65,"userId":"normal"},{"id":67,"userId":"normal"},{"id":68,"userId":"normal"},{"id":69,"userId":"normal"},{"id":70,"userId":"normal"},{"id":71,"userId":"normal"},
 {"id":72,"userId":"normal"},{"id":74,"userId":"normal"},{"id":75,"userId":"normal"},{"id":76,"userId":"normal"},{"id":77,"userId":"normal"},{"id":78,"userId":"normal"},{"id":80,"userId":"normal"}]}
2020-12-09 17:00:02,371 - Nextcloud News Updater - INFO - Updating feed with id 80 and user normal
2020-12-09 17:00:02,372 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=80
2020-12-09 17:00:02,372 - Nextcloud News Updater - INFO - Updating feed with id 78 and user normal
2020-12-09 17:00:02,372 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=78
2020-12-09 17:00:02,372 - Nextcloud News Updater - INFO - Updating feed with id 77 and user normal
2020-12-09 17:00:02,373 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=77
2020-12-09 17:00:02,373 - Nextcloud News Updater - INFO - Updating feed with id 76 and user normal
2020-12-09 17:00:02,373 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=76
2020-12-09 17:00:02,373 - Nextcloud News Updater - INFO - Updating feed with id 75 and user normal
2020-12-09 17:00:02,374 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=75
2020-12-09 17:00:02,374 - Nextcloud News Updater - INFO - Updating feed with id 74 and user normal
2020-12-09 17:00:02,374 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=74
2020-12-09 17:00:02,375 - Nextcloud News Updater - INFO - Updating feed with id 72 and user normal
2020-12-09 17:00:02,375 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=72
2020-12-09 17:00:02,375 - Nextcloud News Updater - INFO - Updating feed with id 71 and user normal
2020-12-09 17:00:02,376 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=71
2020-12-09 17:00:02,376 - Nextcloud News Updater - INFO - Updating feed with id 70 and user normal
2020-12-09 17:00:02,376 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=70
2020-12-09 17:00:02,377 - Nextcloud News Updater - INFO - Updating feed with id 69 and user normal
2020-12-09 17:00:02,377 - Nextcloud News Updater - INFO - Calling update url: http://localhost:8889/index.php/apps/news/api/v1-2/feeds/update?userId=normal&feedId=69
2020-12-09 17:00:02,408 - Nextcloud News Updater - ERROR - HTTP Error 500: Internal Server Error
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/nextcloud_news_updater/api/updater.py", line 34, in run
    self.update_feed(feed)
  File "/usr/lib/python3.6/site-packages/nextcloud_news_updater/api/web.py", line 112, in update_feed
    self.client.get(url, self.auth, self.config.timeout)
  File "/usr/lib/python3.6/site-packages/nextcloud_news_updater/api/web.py", line 61, in get
    response = urlopen(req, timeout=timeout)
  File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

v2:

2020-12-09 17:30:02,863 - Nextcloud News Updater - ERROR - Could not parse given JSON: <!DOCTYPE html>
<html class="ng-csp" data-placeholder-focus="false" lang="en" data-locale="en" >
    <head data-user="[snip]" data-user-displayname="[snip]" data-requesttoken="

v15:

2020-12-09 16:45:01,242 - Nextcloud News Updater - INFO - Running update once with 10 threads
2020-12-09 16:45:01,242 - Nextcloud News Updater - ERROR - 'NoneType' object has no attribute 'before_cleanup_url': Trying again in 30 seconds
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/nextcloud_news_updater/api/updater.py", line 69, in run
    self.before_update()
  File "/usr/lib/python3.6/site-packages/nextcloud_news_updater/api/web.py", line 75, in before_update
    'Calling before update url:  %s' % self.api.before_cleanup_url)
AttributeError: 'NoneType' object has no attribute 'before_cleanup_url'
nRaecheR commented 3 years ago

Same here, I get a lot of errors in NC log like 

[index] Error: Exception: Call to undefined method OCA\News\Db\FeedMapperV2::find() at <<closure>>

0. /var/www/html/lib/private/AppFramework/App.php line 152
   OC\AppFramework\Http\Dispatcher->dispatch(OCA\News\Controller\FeedApiController {}, "update")
1. /var/www/html/lib/private/Route/Router.php line 308
   OC\AppFramework\App::main("OCA\\News\\Cont ... r", "update", OC\AppFramework\ ... {}, {action: null,_r ... "})
2. /var/www/html/lib/base.php line 1008
   OC\Route\Router->match("/apps/news/api/v1-2/feeds/update")
3. /var/www/html/index.php line 37
   OC::handleRequest()

GET /index.php/apps/news/api/v1-2/feeds/update?userId=<userid>&feedId=3
from <ipaddress> by <userid> at 2020-12-09T19:02:52+00:00
BernhardPosselt commented 3 years ago

@DriverXX can you take a look?

freaktechnik commented 3 years ago

Based on the PR it seems I should have my config set to v15, it was just missing the implementation for the REST API based updater path.