search

YunoHost-Apps / wallabag2_ynh

Wallabag v2 package for YunoHost
https://www.wallabag.org/
GNU Affero General Public License v3.0
62 stars 14 forks source link

Redis Pocket import doesn't work #30

Closed movanet closed 5 years ago

movanet commented 7 years ago

Environment: VPS, debian-8.0-x86_64, Memory 768MB, 1 Core. I have opened redis port 6379 on firewall, run redis-server. enable redis on internal setting (set to 1), entered pocket consumer key, entered the bin/console wallabag:import:redis-worker -e=prod pocket -vv >> /path/to/wallabag/var/logs/redis-pocket.log on redis server (path changed in accordance to my config). It send "Messages in Queue 1681" on the import page and "basic save xxxx"on SSH. However, nothing happened and it has been 8 hours.

It was working fine previously. Any ideas?

JimboJoe commented 7 years ago

On what version did you test? On this pending PR? On what version of YunoHost package and Wallabag was it previously working? Do you have any interesting log inputs in /path/to/wallabag/var/logs/redis-pocket.log, /var/log/redis/redis-server.log or /var/www/wallabag2/var/logs/prod.log? How did did you execute precisely your bin/console command (what user)?

I must admit this issue will be difficult to investigate as I have no Pocket account...

movanet commented 7 years ago

@JimboJoe

  1. I have Version: 2.2.2
  2. Same version, I think
  3. /var/www/wallabag2/var/logs is empty
  4. redis-server.log is below.
  5. I tried setting vm.overcommit_memory = 1 in /etc/sysctl.conf and reboot but to no avail. Running sysctl vm.overcommit_memory=1 result in permission denied
  6. I am running redis-server as root. Wallabag user is similar to Yunohost user (only 1 single user as admin).
  7. Yunohost is only running wallabag v2, no other applications are installed

[18674] 20 Apr 10:03:58.218 # You requested maxclients of 10000 requiring at least 10032 max file descriptors. [18674] 20 Apr 10:03:58.218 # Redis can't set maximum open files to 10032 because of OS error: Operation not permitted. [18674] 20 Apr 10:03:58.218 # Current maximum open files is 1024. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'. .
_.-__ ''-._ _.- .. ''-. Redis 2.8.17 (00000000/0) 64 bit .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in stand alone mode |`-._`-...-` __...-.-.|'` .-'| Port: 6379 | -._. / .-' | PID: 18674 -._-. `-./ .-' .-'
|`-.-._-..-' .-'.-'|
| -._-. .-'.-' | http://redis.io
`-. -._-..-'.-' .-'
|-._-._ -.__.-' _.-'_.-'| |-.`-. .-'.-' |
-._-._-.__.-'_.-' _.-' -._ -.__.-' _.-' -. .-'
`-.__.-'

[18674] 20 Apr 10:03:58.220 # Server started, Redis version 2.8.17 [18674] 20 Apr 10:03:58.220 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. [18674] 20 Apr 10:03:58.220 The server is now ready to accept connections on port 6379 [18674] 20 Apr 10:14:46.809 10 changes in 300 seconds. Saving... [18674] 20 Apr 10:14:46.810 Background saving started by pid 11029 [11029] 20 Apr 10:14:46.859 DB saved on disk [11029] 20 Apr 10:14:46.859 RDB: 0 MB of memory used by copy-on-write [18674] 20 Apr 10:14:46.910 Background saving terminated with success [18674] 20 Apr 10:19:47.060 10 changes in 300 seconds. Saving... [18674] 20 Apr 10:19:47.060 Background saving started by pid 11045 [11045] 20 Apr 10:19:47.086 DB saved on disk [11045] 20 Apr 10:19:47.086 RDB: 0 MB of memory used by copy-on-write [18674] 20 Apr 10:19:47.160 * Background saving terminated with success

movanet commented 7 years ago

Set LimitNOFILE = 65000. Received no more maxclients report on redis server. However, no changes on import.

lapineige commented 6 years ago

Does it work now ? (with a more recent version)

mediaformat commented 3 years ago

Was this ever fixed? I am having a similar issue, Redis is not showing any activity. Wallabag says I have ~1000 entries in queue

lapineige commented 3 years ago

Can you provide some log ? Redis service status ?

mediaformat commented 3 years ago

The redis-server status is: Running These are the logs since trying to async import with Redis:

733:M 26 Jan 2021 11:16:44.027 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 11:16:44.031 Background saving started by pid 2443 2443:C 26 Jan 2021 11:16:44.128 DB saved on disk 2443:C 26 Jan 2021 11:16:44.129 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 11:16:44.131 Background saving terminated with success 733:M 26 Jan 2021 11:21:45.075 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 11:21:45.078 Background saving started by pid 2492 2492:C 26 Jan 2021 11:21:45.217 DB saved on disk 2492:C 26 Jan 2021 11:21:45.219 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 11:21:45.279 Background saving terminated with success 733:M 26 Jan 2021 11:26:46.077 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 11:26:46.079 Background saving started by pid 2543 2543:C 26 Jan 2021 11:26:46.216 DB saved on disk 2543:C 26 Jan 2021 11:26:46.218 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 11:26:46.281 Background saving terminated with success 733:M 26 Jan 2021 11:31:47.011 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 11:31:47.021 Background saving started by pid 2578 2578:C 26 Jan 2021 11:31:47.155 DB saved on disk 2578:C 26 Jan 2021 11:31:47.157 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 11:31:47.223 Background saving terminated with success 733:M 26 Jan 2021 11:36:48.079 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 11:36:48.080 Background saving started by pid 2784 2784:C 26 Jan 2021 11:36:48.299 DB saved on disk 2784:C 26 Jan 2021 11:36:48.301 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 11:36:48.382 Background saving terminated with success 733:M 26 Jan 2021 11:41:49.101 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 11:41:49.105 Background saving started by pid 2880 2880:C 26 Jan 2021 11:41:49.220 DB saved on disk 2880:C 26 Jan 2021 11:41:49.221 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 11:41:49.307 Background saving terminated with success 733:M 26 Jan 2021 11:46:50.027 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 11:46:50.032 Background saving started by pid 2929 2929:C 26 Jan 2021 11:46:50.180 DB saved on disk 2929:C 26 Jan 2021 11:46:50.181 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 11:46:50.234 Background saving terminated with success 733:M 26 Jan 2021 11:51:51.056 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 11:51:51.059 Background saving started by pid 3004 3004:C 26 Jan 2021 11:51:51.203 DB saved on disk 3004:C 26 Jan 2021 11:51:51.204 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 11:51:51.260 Background saving terminated with success 733:M 26 Jan 2021 11:56:52.028 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 11:56:52.030 Background saving started by pid 3102 3102:C 26 Jan 2021 11:56:52.149 DB saved on disk 3102:C 26 Jan 2021 11:56:52.150 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 11:56:52.231 Background saving terminated with success 733:M 26 Jan 2021 12:01:53.011 10 changes in 300 seconds. Saving... 733:M 26 Jan 2021 12:01:53.013 Background saving started by pid 3175 3175:C 26 Jan 2021 12:01:53.146 DB saved on disk 3175:C 26 Jan 2021 12:01:53.148 RDB: 0 MB of memory used by copy-on-write 733:M 26 Jan 2021 12:01:53.214 Background saving terminated with success

mediaformat commented 3 years ago

Looking at /app/config/parameters.yml

    database_driver: pdo_mysql
    database_driver_class: ~
    # database_driver_class: Wallabag\CoreBundle\Doctrine\DBAL\Driver\CustomPostgreSQLDriver
    database_host: 127.0.0.1
    database_port: ~
    database_name:  wallabag2
    database_user: ****
    database_password: ****
    # For SQLite, database_path should be "%kernel.project_dir%/data/db/wallabag.sqlite"
    database_path: null
    database_table_prefix: null
    database_socket: null
    # with PostgreSQL and SQLite, you must set "utf8"
    database_charset: utf8mb4

    domain_name: https://wallabag.myrealdomainhere.tld/

   # Redis processing
    redis_scheme: tcp
    redis_host: localhost
    redis_port: 6379
    redis_path: null
    redis_password: null

Wondering if the Redis processing section has correct info

mediaformat commented 3 years ago

I ended up doing it from the command line

# for Pocket import
bin/console wallabag:import:redis-worker --env=prod pocket -vv >> /path/to/wallabag/var/logs/redis-pocket.log

But this practically crashed my server, for about 1000 items in my pocket library it took around 6-8 hours (w 2gb of RAM).

lapineige commented 3 years ago

That's very strange… My last import, for about the same amount of articles, took less than half an hour… (1GB RAM)

mediaformat commented 3 years ago

@lapineige did you have to initiate the import from the command line, or did it just work from within wallabag?

lapineige commented 3 years ago

It worked in Wallabag directly.