search

nextcloud / news

:newspaper: RSS/Atom feed reader
https://apps.nextcloud.com/apps/news
GNU Affero General Public License v3.0
866 stars 186 forks source link

News Feeds not updated with NC12 #155

Closed crequill closed 7 years ago

crequill commented 7 years ago

After Nexcloud 12 update, there is no more news feeds updates.

System Information

Contents of nextcloud/data/nextcloud.log

{"reqId":"fpp","level":0,"time":"2017-05-23T14:00:03+00:00","remoteAddr":"","user":"--","app":"cron","method":"--","url":"--","message":"Run OC\\BackgroundJob\\Legacy\\RegularJob job with ID 16","userAgent":"--","version":"12.0.0.29"}
{"reqId":"fpp","level":0,"time":"2017-05-23T14:00:03+00:00","remoteAddr":"","user":"--","app":"cron","method":"--","url":"--","message":"Finished OC\\BackgroundJob\\Legacy\\RegularJob job with ID 16 in 0 seconds","userAgent":"--","version":"12.0.0.29"}
{"reqId":"fpp","level":0,"time":"2017-05-23T14:00:04+00:00","remoteAddr":"","user":"--","app":"cron","method":"--","url":"--","message":"Run OC\\Log\\Rotate job with ID 20","userAgent":"--","version":"12.0.0.29"}
{"reqId":"fpp","level":0,"time":"2017-05-23T14:00:04+00:00","remoteAddr":"","user":"--","app":"cron","method":"--","url":"--","message":"Finished OC\\Log\\Rotate job with ID 20 in 0 seconds","userAgent":"--","version":"12.0.0.29"}

Problem description

cron is working. There is no errors in log. useCronUpdates = true in nextcloud/data/news/config/config.ini I use the system cron (Cron) setting. The crontab exist and there is no problem of file permissions. I run php -f nextcloud/cron.php, there is no error but after adding an error_log call in the background job file, I don’t see it in php errors log. The news Updater seems not called by nextcloud cron. This was working fine with NC11. Any idea?

ndurchx commented 7 years ago

+1

koarlmichael commented 7 years ago

Same here +1

tobiasKaminsky commented 7 years ago

Just a little workaround: php ./occ news:updater:all-feeds to collect all ids and userIds and then a bash script for invoking news:updater:update-feed $id $userId

tobiasKaminsky commented 7 years ago

Bernhard told me on IRC to disable and re-enable the app. I will have a look if this is enough.

SchnorcherSepp commented 7 years ago

same here, no news updates :*(

crequill commented 7 years ago

Yes it works. Thanks. Disable and re enable the app and feeds are updated again.

michaelpjohnson commented 7 years ago

Disabling and re-enabling only partially worked. Some of my feeds are updating and others are not.

ghost commented 7 years ago

I also have no updates. I just disabled/enabled, I'll report back if it worked

EDIT: does seem to work. All my feeds have new stuff

mweimerskirch commented 7 years ago

Same problem. Disabling and re-enabling the app worked.

nickvergessen commented 7 years ago

I'm looking into the issue today

michaelpjohnson commented 7 years ago

@nickvergessen Thanks, for looking into this. I'm working from home today, so if you need me to try anything or upload my config etc. let me know. None of my feeds are updating.

As a last ditch effort I was going to export my OPML, reset the DB and reinstall. But I'll wait to see what you think.

Steps I Took

$ sudo -u www-data ./occ app:disable news
news disabled
$ sudo -u www-data ./occ app:enable news
news enabled
$ sudo -u www-data php -f /srv/www/nextcloud/cron.php

My Env

sudo -u www-data ./occ status
  - installed: true
  - version: 12.0.0.29
  - versionstring: 12.0.0
  - edition: 
$ php --version
PHP 7.0.15-0ubuntu0.16.04.4 (cli) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.0.15-0ubuntu0.16.04.4, Copyright (c) 1999-2017, by Zend Technologies

$ apache2 -version
Server version: Apache/2.4.18 (Ubuntu)
Server built:   2017-05-05T16:32:00

$ mysql --version
mysql  Ver 15.1 Distrib 10.0.29-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
nickvergessen commented 7 years ago

Fix is at: https://github.com/nextcloud/server/pull/5090

To test, apply the patch, reduce the version number of the news app in oc_appconfig, and then visit your instance to run the app update.

michaelpjohnson commented 7 years ago

@nickvergessen This is what I ran: 

> update oc_appconfig set configvalue='11.0.0' where appid='news' and configkey='installed_version';

but after the upgrade, this is what I'm seeing: Some files have not passed the integrity check. Further information on how to resolve this issue can be found in our documentation. (List of invalid files… / Rescan…)

michaelpjohnson commented 7 years ago

I think it's because I applied it by hand:

$ sudo -u www-data php occ integrity:check-core
  - INVALID_HASH:
    - lib/private/legacy/app.php:
      - expected: a752283699a15deaced52fbf9584d9b5c5da4845ba5c0ba8ee24550288e42d00549905885df943eb61756137130578a4f7dd8fe14475b702c97ccea9341b32e3
      - current: 5963753b56f23bb4b94936b8332082718e6d0c09dc298a712191491980038c1589a66866ed0eebfcab427d9dfaf8960fe3dccc3fce19e15021d10eaca0fbc567
nickvergessen commented 7 years ago

yeah, you will have to live with that until the next update, or you revert the patch.

michaelpjohnson commented 7 years ago

@nickvergessen Oh in that case then it didn't seem to work. Many of my feeds are failing (some feeds are updating, but not all)

Here's an example for XKCD: http://xkcd.com/rss.xml 


$ sudo -u www-data ./occ news:updater:update-feed 68 mjohnson

SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\xF0\x9F\x91\x8F" ...' for column 'body' at row 1 
Could not update feed with id 68 and user mjohnson: An exception occurred while executing 'INSERT INTO `oc_news_items`(```
nickvergessen commented 7 years ago

Well that is a different issue.

If one updated, my patch worked. Everything else is unrelated.

michaelpjohnson commented 7 years ago

@nickvergessen OK I see, I thought you were fixing the bug in the fetching, but it looks like you fixed the enable/disable bug. I was always able to fetch some feeds. But NC12 broke fetching of the feeds themselves which is what I thought this ticket was. I'll open a new one to address the broken feeds and the SQL errors above. Thanks.

BernhardPosselt commented 7 years ago

@michaelpjohnson check any of these issues, your database is configured incorrectly https://github.com/nextcloud/news/issues?q=is%3Aissue+utf8mb4+is%3Aclosed

michaelpjohnson commented 7 years ago

Thanks @BernhardPosselt I'll change my DB

BernhardPosselt commented 7 years ago

@michaelpjohnson the 11.0.3 release will warn you (at least for database configuration, however not for columns and tables) and display a link to solve the schema issue when you open the app

michaelpjohnson commented 7 years ago

@BernhardPosselt I followed this https://docs.nextcloud.com/server/11/admin_manual/maintenance/mysql_4byte_support.html and it works perfectly now.

SchnorcherSepp commented 7 years ago

Hi! How does the charset check exactly work?

My database already has the correct charset:

+--------------+--------------------+----------------------------+------------------------+----------+
| CATALOG_NAME | SCHEMA_NAME        | DEFAULT_CHARACTER_SET_NAME | DEFAULT_COLLATION_NAME | SQL_PATH |
+--------------+--------------------+----------------------------+------------------------+----------+
| def          | information_schema | utf8                       | utf8_general_ci        | NULL     |
| def          | mysql              | utf8mb4                    | utf8mb4_general_ci     | NULL     |
| def          | nextcloud          | utf8mb4                    | utf8mb4_general_ci     | NULL     |
| def          | performance_schema | utf8                       | utf8_general_ci        | NULL     |
+--------------+--------------------+----------------------------+------------------------+----------+

aaand my tables

+---------------+--------------------+----------------+-------------+--------------------+---------+------------+------------+----------------+-------------+--------------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+--------------------+---------------------------------------------------+-------------------+--------------------+
| TABLE_CATALOG | TABLE_SCHEMA       | TABLE_NAME     | TABLE_TYPE  | ENGINE             | VERSION | ROW_FORMAT | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH | MAX_DATA_LENGTH    | INDEX_LENGTH | DATA_FREE | AUTO_INCREMENT | CREATE_TIME         | UPDATE_TIME         | CHECK_TIME          | TABLE_COLLATION   | CHECKSUM | CREATE_OPTIONS     | TABLE_COMMENT                                     | COLLATION_NAME    | CHARACTER_SET_NAME |
+---------------+--------------------+----------------+-------------+--------------------+---------+------------+------------+----------------+-------------+--------------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+--------------------+---------------------------------------------------+-------------------+--------------------+
| def           | nextcloud          | news_feeds     | BASE TABLE  | InnoDB             |      10 | Compact    |         22 |            744 |       16384 |                  0 |        65536 |         0 |             41 | 2017-05-25 11:50:30 | NULL                | NULL                | utf8mb4_bin       |     NULL |                    |                                                   | utf8mb4_bin       | utf8mb4            |
| def           | nextcloud          | news_folders   | BASE TABLE  | InnoDB             |      10 | Compact    |          3 |           5461 |       16384 |                  0 |        49152 |         0 |              7 | 2017-05-25 11:50:41 | NULL                | NULL                | utf8mb4_bin       |     NULL |                    |                                                   | utf8mb4_bin       | utf8mb4            |
| def           | nextcloud          | news_items     | BASE TABLE  | InnoDB             |      10 | Compact    |       2317 |           3401 |     7880704 |                  0 |       655360 |   4194304 |           5855 | 2017-05-25 11:50:52 | NULL                | NULL                | utf8mb4_bin       |     NULL |                    |                                                   | utf8mb4_bin       | utf8mb4            |
+---------------+--------------------+----------------+-------------+--------------------+---------+------------+------------+----------------+-------------+--------------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+--------------------+---------------------------------------------------+-------------------+--------------------+

but i see the warning in the news app :(

zwischenablage02

michaelpjohnson commented 7 years ago

@SchnorcherSepp Did you also update your my.cnf?

These are the steps I took: (from https://docs.nextcloud.com/server/11/admin_manual/maintenance/mysql_4byte_support.html)

[mysqld]
innodb_large_prefix=on
innodb_file_format=barracuda
innodb_file_per_table=1

Restart the MySQL server in case you changed the configuration in step 2.

Change your databases character set and collation:

ALTER DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; Set the mysql.utf8mb4 config to true in your config.php:

$ sudo -u www-data occ config:system:set mysql.utf8mb4 --type boolean --value="true" Convert all existing tables to the new collation by running the repair step:

$ sudo -u www-data occ maintenance:repair

The last two lines are what cleared the warning for me.

koarlmichael commented 7 years ago

@michaelpjohnson, its not working for me after sudo -u www-data php occ maintenance:repair

[Doctrine\DBAL\Exception\DriverException] An exception occurred while executing 'ALTER TABLEoc_addressbooksCONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;': SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes [Doctrine\DBAL\Driver\PDOException] SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes [PDOException] SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes

michaelpjohnson commented 7 years ago

@koarlmichael I got that as well. All I did was make sure that 

[mysqld]
innodb_large_prefix=on
innodb_file_format=barracuda
innodb_file_per_table=1

was in my /etc/mysql/my.cnf and then I restarted mysql.

mweimerskirch commented 7 years ago

@michaelpjohnson and @koarlmichael "innodb_large_prefix" was introduced with MySQL 5.6.3, so if you have an older version that won't work because the keys are limited in length and you'll have to stay with regular utf8 instead of utf8mb4. I think MariaDB has supported this since 10.0, but I'm not 100% sure (I'm running MariaDB 10.2 which supports the "large_prefix" option).

What I don't understand is why the news app requires utf8mb4. Is that really necessary?

wwwindisch commented 7 years ago

@koarlmichael, @michaelpjohnson The problem is caused by the wrong length of the index keys of existing columns for utf8mb4. I ended up with the same issue reproducing your steps. See

for more information on that.

I've solved the problem by adjusting the character length of the keys. Please note, that I'm a complete noob regarding mysql and don't really know in detail, what I was doing ;-) It's just googled information put together... My NextCloud is still alive and from a first look at it fully functional after applying the following steps...but there is no warranty. And grab some coffee...it will take a while...

  1. A backup of your database would be nice...but since you are already run into errors it's probably too late anyway :-/

  2. Log in to your MySQL Database:

    • Open Shell (best in a separate instance (B); keep the one where you call sudo -u www-data php occ maintenance:repair open in parallel (A)) and become root: sudo su
    • MySQL login: mysql -u DBUSER-p DBNAME --> You can find the credentials (DBUSER, DBNAME and Password) in your NextCloud config: /var/www/nexcloud/config/config.php

  3. Fix a table

    • Open the table, which causes errors (in you case oc_addressbooks) in your MySQL-Shell (B): SHOW FULL COLUMNS FROM oc_addressbooks;
    • It will show a table. Important are the columns "Field" and "Type". You need to change all Fields with a vchar-Type larger than 191. Since we want to convert from 3 bit to 4 bit, all lengths have to be multiplied by 0.75 (rounded to integer). That means: 255 --> 191, 512 --> 384
    • Changeing the length works like that (for the table _ocaddressbooks and the field principaluri): ALTER TABLE oc_addressbooks MODIFY principaluri VARCHAR(191) CHARACTER SET utf8mb4;
    • Repeat the previous step for all fields in the table, which are too long.

  4. Find the next problematic table

    • Switch to your other shell (A) and run sudo -u www-data php occ maintenance:repair (or sudo -u www-data php /var/www/nextcloud/occ maintenance:repair) again
    • Wait for the next error and repeat step 2 for als fields in this table.
    • Repeat the steps 2 and 3 till you get a clean run of maintenance:repair.

You'll get error messages like "Failed to load repair step for ..." when running maintenance:repair. This is because NextCloud is left in maintenance mode, when the repair function crashes. In this mode Apps are not loaded and hence, their repair steps can't be found. Therefore finalize with the following steps: sudo -u www-data php /var/www/nextcloud/occ maintenance:mode --off sudo -u www-data php /var/www/nextcloud/occ maintenance:repair --> This very last run should now (hopefully) pass without errors.

Finally, my News App is happy with the DB and ... I hope it was worth it :-/ Good luck & fingers crossed!

EDIT: The whole procedure was necessary because I'm running MySQL 5.5.55. According to @mweimerskirch it might not be necessary on newer versions.

BernhardPosselt commented 7 years ago

@wwwindisch it is required because feeds include emojis nowadays and that will break your feed updates

BernhardPosselt commented 7 years ago

This specific issue is solved. Use a new topic for new issues