search

turt2live / matrix-dimension

An open source integration manager for matrix clients, like Element.
https://dimension.t2bot.io
GNU General Public License v3.0
433 stars 110 forks source link

Importing stickers from Telegram doesn't work anymore #364

Open beardedlinuxgeek opened 4 years ago

beardedlinuxgeek commented 4 years ago

309 should be reopened, it was never fixed. Multiple users reporting they still get the 404 error. I can also confirm that importing from telegram does not work and I am getting 404s.

[3992]: Jul-6-2020 01:34:52.539 +00:00 info [Webserver] Incoming request: POST /api/v1/dimension/admin/stickers/packs/import/telegram?scalar_token=redacted
[3992]: Jul-6-2020 01:34:52.541 +00:00 warn [MatrixSecurity] Request used old scalar_token auth - this will be removed in a future version
[3992]: Jul-6-2020 01:34:52.543 +00:00 info [AdminStickerService] Importing https://t.me/addstickers/facebookfoxes from Telegram for @beardedlinuxgeek:example.com
[3992]: Error: 404: Not Found
[3992]:     at buildConfig.then.then.then.then (/home/node/matrix-dimension/node_modules/telegraf/core/network/client.js:263:17)
[3992]:     at process._tickCallback (internal/process/next_tick.js:68:7)
Craeckie commented 4 years ago

There seems to be a difference between static and animated stickers. I'm able to add these: https://t.me/addstickers/MW001 and https://t.me/addstickers/ClintonVSTrump When I click on "Import from Telegram" nothing happens. But after waiting a few seconds and re-opening the window, the sticker-set appears.

But the animated ones don't work, e.g. https://t.me/addstickers/BreadToast or https://t.me/addstickers/MooingCow It says "Error importing sticker pack".

The version shown in the integration manager is v1.0.0-db22981.

asmaps commented 3 years ago

Any news here? It definitely seems to break on animated stickers. "HotCherry" (animated) does not work, "Blahaj" (static) is working fine.

https://t.me/addstickers/HotCherry

https://t.me/addstickers/Blahaj

dimension_1         | Jan-3-2021 16:58:21.097 +00:00 info [Webserver] Incoming request: POST /api/v1/dimension/admin/stickers/packs/import/telegram?scalar_token=redacted
dimension_1         | Jan-3-2021 16:58:21.098 +00:00 warn [MatrixSecurity] Request used old scalar_token auth - this will be removed in a future version
dimension_1         | Jan-3-2021 16:58:21.100 +00:00 info [AdminStickerService] Importing https://t.me/addstickers/HotCherry from Telegram for @abc:example.com
dimension_1         | Jan-3-2021 16:58:21.625 +00:00 info [AdminStickerService] Importing sticker from https://api.telegram.org/file/bot******:*******/stickers/file_0.tgs
dimension_1         | Jan-3-2021 16:58:21.740 +00:00 error [AdminStickerService] Input buffer contains unsupported image format
dimension_1         | Error: Input buffer contains unsupported image format
dimension_1         | Jan-3-2021 16:58:21.757 +00:00 warn [Webserver] Handling ApiError 500 {"message":"Error importing stickers","dim_errcode":"M_UNKNOWN","error":"Error importing stickers","errcode":"M_UNKNOWN"}
razorness commented 3 years ago 
Jan 17 04:42:31 matrix-dimension[3455]: Jan-17-2021 03:42:31.932 +00:00 info [Webserver] Incoming request: POST /api/v1/dimension/stickers/packs/import?scalar_token=redacted
Jan 17 04:42:31 matrix-dimension[3455]: Jan-17-2021 03:42:31.932 +00:00 warn [MatrixSecurity] Request used old scalar_token auth - this will be removed in a future version
Jan 17 04:42:31 matrix-dimension[3455]: Jan-17-2021 03:42:31.939 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "id", "userId", "scalarToken", "isDimensionToken", "upstreamId" FROM "dimension_scalar_tokens" AS "UserScalarToken" WHERE "UserScalarToken"."userId" = '@user:domain';
Jan 17 04:42:31 matrix-dimension[3455]: Jan-17-2021 03:42:31.941 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "id", "name", "type", "scalarUrl", "apiUrl" FROM "dimension_upstreams" AS "Upstream";
Jan 17 04:42:31 matrix-dimension[3455]: Jan-17-2021 03:42:31.943 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "UserScalarToken"."id", "UserScalarToken"."userId", "UserScalarToken"."scalarToken", "UserScalarToken"."isDimensionToken", "UserScalarToken"."upstreamId", "upstream"."id" AS "upstream.id", "upstream"."name" AS "upstream.name", "upstream"."type" AS "upstream.type", "upstream"."scalarUrl" AS "upstream.scalarUrl", "upstream"."apiUrl" AS "upstream.apiUrl" FROM "dimension_scalar_tokens" AS "UserScalarToken" LEFT OUTER JOIN "dimension_upstreams" AS "upstream" ON "UserScalarToken"."upstreamId" = "upstream"."id" WHERE "UserScalarToken"."isDimensionToken" = false AND "UserScalarToken"."userId" = '@dimension:razorness.de' AND "UserScalarToken"."upstreamId" = 1;
Jan 17 04:42:31 matrix-dimension[3455]: Jan-17-2021 03:42:31.944 +00:00 info [ScalarClient] Doing upstream scalar request: https://scalar.vector.im/api/account
Jan 17 04:42:32 matrix-dimension[3455]: Jan-17-2021 03:42:32.063 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "id", "termsId", "userId" FROM "dimension_terms_signed" AS "TermsSignedRecord" WHERE "TermsSignedRecord"."userId" = '@user:domain';
Jan 17 04:42:32 matrix-dimension[3455]: Jan-17-2021 03:42:32.065 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "UserScalarToken"."id", "UserScalarToken"."userId", "UserScalarToken"."scalarToken", "UserScalarToken"."isDimensionToken", "UserScalarToken"."upstreamId", "upstream"."id" AS "upstream.id", "upstream"."name" AS "upstream.name", "upstream"."type" AS "upstream.type", "upstream"."scalarUrl" AS "upstream.scalarUrl", "upstream"."apiUrl" AS "upstream.apiUrl" FROM "dimension_scalar_tokens" AS "UserScalarToken" LEFT OUTER JOIN "dimension_upstreams" AS "upstream" ON "UserScalarToken"."upstreamId" = "upstream"."id" WHERE "UserScalarToken"."userId" = '@user:domain';
Jan 17 04:42:32 matrix-dimension[3455]: Jan-17-2021 03:42:32.068 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "UserScalarToken"."id", "UserScalarToken"."userId", "UserScalarToken"."scalarToken", "UserScalarToken"."isDimensionToken", "UserScalarToken"."upstreamId", "upstream"."id" AS "upstream.id", "upstream"."name" AS "upstream.name", "upstream"."type" AS "upstream.type", "upstream"."scalarUrl" AS "upstream.scalarUrl", "upstream"."apiUrl" AS "upstream.apiUrl" FROM "dimension_scalar_tokens" AS "UserScalarToken" LEFT OUTER JOIN "dimension_upstreams" AS "upstream" ON "UserScalarToken"."upstreamId" = "upstream"."id" WHERE "UserScalarToken"."isDimensionToken" = false AND "UserScalarToken"."userId" = '@dimension:razorness.de' AND "UserScalarToken"."upstreamId" = 1;
Jan 17 04:42:32 matrix-dimension[3455]: Jan-17-2021 03:42:32.069 +00:00 info [ScalarClient] Doing upstream scalar request: https://scalar.vector.im/_matrix/integrations/v1/account
Jan 17 04:42:32 matrix-dimension[3455]: Jan-17-2021 03:42:32.260 +00:00 info [ScalarClient] Doing upstream scalar request: https://scalar.vector.im/_matrix/integrations/v1/account
Jan 17 04:42:32 matrix-dimension[3455]: Error: Failed to locate Dimension metadata
Jan 17 04:42:32 matrix-dimension[3455]:     at /home/node/matrix-dimension/build/app/utils/StickerpackMetadataDownloader.js:14:23
Jan 17 04:42:32 matrix-dimension[3455]:     at tryCatcher (/home/node/matrix-dimension/node_modules/bluebird/js/release/util.js:16:23)
Jan 17 04:42:32 matrix-dimension[3455]:     at Promise._settlePromiseFromHandler (/home/node/matrix-dimension/node_modules/bluebird/js/release/promise.js:517:31)
Jan 17 04:42:32 matrix-dimension[3455]:     at Promise._settlePromise (/home/node/matrix-dimension/node_modules/bluebird/js/release/promise.js:574:18)
Jan 17 04:42:32 matrix-dimension[3455]:     at Promise._settlePromise0 (/home/node/matrix-dimension/node_modules/bluebird/js/release/promise.js:619:10)
Jan 17 04:42:32 matrix-dimension[3455]:     at Promise._settlePromises (/home/node/matrix-dimension/node_modules/bluebird/js/release/promise.js:699:18)
Jan 17 04:42:32 matrix-dimension[3455]:     at _drainQueueStep (/home/node/matrix-dimension/node_modules/bluebird/js/release/async.js:138:12)
Jan 17 04:42:32 matrix-dimension[3455]:     at _drainQueue (/home/node/matrix-dimension/node_modules/bluebird/js/release/async.js:131:9)
Jan 17 04:42:32 matrix-dimension[3455]:     at Async._drainQueues (/home/node/matrix-dimension/node_modules/bluebird/js/release/async.js:147:5)
Jan 17 04:42:32 matrix-dimension[3455]:     at Immediate.Async.drainQueues [as _onImmediate] (/home/node/matrix-dimension/node_modules/bluebird/js/release/async.js:17:14)
Jan 17 04:42:32 matrix-dimension[3455]:     at processImmediate (internal/timers.js:456:21)

Definitly can't add any sticker pack ...

Running current docker image: turt2live/matrix-dimension:latest

Tried listed aboth and others:

TheTimeWalker commented 3 years ago

@razorness Note that the Hot Cherry pack has animated stickers. They generally won't work and are for a different issue: #361

xthursdayx commented 3 years ago

I'm getting basically the same error messages as @razorness when trying to import non-animated sticker packs:

Jan 19 16:30:15 synapse matrix-dimension[1483]: Jan-19-2021 16:30:15.545 +00:00 info [Webserver] Incoming request: POST /api/v1/dimension/stickers/packs/import?scalar_token=redacted
Jan 19 16:30:15 synapse matrix-dimension[1483]: Jan-19-2021 16:30:15.546 +00:00 warn [MatrixSecurity] Request used old scalar_token auth - this will be removed in a future version
Jan 19 16:30:15 synapse matrix-dimension[1483]: Jan-19-2021 16:30:15.553 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "id", "userId", "scalarToken", "isDimensionToken", "upstreamId" FROM "dimension_scalar_tokens" AS "UserScalarToken" WHERE "UserScalarToken"."userId" = '@user:domain';
Jan 19 16:30:15 synapse matrix-dimension[1483]: Jan-19-2021 16:30:15.556 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "id", "name", "type", "scalarUrl", "apiUrl" FROM "dimension_upstreams" AS "Upstream";
Jan 19 16:30:15 synapse matrix-dimension[1483]: Jan-19-2021 16:30:15.558 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "id", "termsId", "userId" FROM "dimension_terms_signed" AS "TermsSignedRecord" WHERE "TermsSignedRecord"."userId" = '@zandrsn:sublm.nl';
Jan 19 16:30:15 synapse matrix-dimension[1483]: Jan-19-2021 16:30:15.561 +00:00 verbose [DimensionStore [SQL]] Executing (default): SELECT "UserScalarToken"."id", "UserScalarToken"."userId", "UserScalarToken"."scalarToken", "UserScalarToken"."isDimensionToken", "UserScalarToken"."upstreamId", "upstream"."id" AS "upstream.id", "upstream"."name" AS "upstream.name", "upstream"."type" AS "upstream.type", "upstream"."scalarUrl" AS "upstream.scalarUrl", "upstream"."apiUrl" AS "upstream.apiUrl" FROM "dimension_scalar_tokens" AS "UserScalarToken" LEFT OUTER JOIN "dimension_upstreams" AS "upstream" ON "UserScalarToken"."upstreamId" = "upstream"."id" WHERE "UserScalarToken"."userId" = '@zandrsn:sublm.nl';
Jan 19 16:30:15 synapse matrix-dimension[1483]: Error: Failed to locate Dimension metadata
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at /home/node/matrix-dimension/build/app/utils/StickerpackMetadataDownloader.js:14:23
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at tryCatcher (/home/node/matrix-dimension/node_modules/bluebird/js/release/util.js:16:23)
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at Promise._settlePromiseFromHandler (/home/node/matrix-dimension/node_modules/bluebird/js/release/promise.js:517:31)
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at Promise._settlePromise (/home/node/matrix-dimension/node_modules/bluebird/js/release/promise.js:574:18)
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at Promise._settlePromise0 (/home/node/matrix-dimension/node_modules/bluebird/js/release/promise.js:619:10)
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at Promise._settlePromises (/home/node/matrix-dimension/node_modules/bluebird/js/release/promise.js:699:18)
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at _drainQueueStep (/home/node/matrix-dimension/node_modules/bluebird/js/release/async.js:138:12)
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at _drainQueue (/home/node/matrix-dimension/node_modules/bluebird/js/release/async.js:131:9)
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at Async._drainQueues (/home/node/matrix-dimension/node_modules/bluebird/js/release/async.js:147:5)
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at Immediate.Async.drainQueues [as _onImmediate] (/home/node/matrix-dimension/node_modules/bluebird/js/release/async.js:17:14)
Jan 19 16:30:15 synapse matrix-dimension[1483]:     at processImmediate (internal/timers.js:456:21)

I'm running Dimension in a Docker container via the matrix-docker-ansible-deploy playbook on an Ubuntu VM, with the host server managing the NGINX reverse proxy, ssl, etc. 'm the NGINX confs created by the ansible playbook, so I assume that they should have the correct SSL and cipher/algorithm settings?

Tupsi commented 3 years ago

Still not working, but got a different error then above (using docker image from today :latest) (Dimension admin page says v1.0.0)

Feb 01 18:51:05 matrix-dimension[16968]: Feb-1-2021 17:51:05.128 +00:00 info [Webserver] Incoming request: POST /api/v1/dimension/admin/stickers/packs/import/telegram?scalar_token=redacted
Feb 01 18:51:05 matrix-dimension[16968]: Feb-1-2021 17:51:05.129 +00:00 warn [MatrixSecurity] Request used old scalar_token auth - this will be removed in a future version
Feb 01 18:51:05 matrix-dimension[16968]: Feb-1-2021 17:51:05.131 +00:00 info [AdminStickerService] Importing https://t.me/addstickers/Blahaj from Telegram for @myuser:mydomain
Feb 01 18:51:05 matrix-dimension[16968]: Error: 404: Not Found
Feb 01 18:51:05 matrix-dimension[16968]:     at /home/node/matrix-dimension/node_modules/telegraf/core/network/client.js:263:17
Feb 01 18:51:05 matrix-dimension[16968]:     at runMicrotasks (<anonymous>)
Feb 01 18:51:05 matrix-dimension[16968]:     at processTicksAndRejections (internal/process/task_queues.js:97:5)
m4rkell commented 3 years ago

Still not working, but got a different error then above (using docker image from today :latest) (Dimension admin page says v1.0.0)

Feb 01 18:51:05 matrix-dimension[16968]: Feb-1-2021 17:51:05.128 +00:00 info [Webserver] Incoming request: POST /api/v1/dimension/admin/stickers/packs/import/telegram?scalar_token=redacted
Feb 01 18:51:05 matrix-dimension[16968]: Feb-1-2021 17:51:05.129 +00:00 warn [MatrixSecurity] Request used old scalar_token auth - this will be removed in a future version
Feb 01 18:51:05 matrix-dimension[16968]: Feb-1-2021 17:51:05.131 +00:00 info [AdminStickerService] Importing https://t.me/addstickers/Blahaj from Telegram for @myuser:mydomain
Feb 01 18:51:05 matrix-dimension[16968]: Error: 404: Not Found
Feb 01 18:51:05 matrix-dimension[16968]:     at /home/node/matrix-dimension/node_modules/telegraf/core/network/client.js:263:17
Feb 01 18:51:05 matrix-dimension[16968]:     at runMicrotasks (<anonymous>)
Feb 01 18:51:05 matrix-dimension[16968]:     at processTicksAndRejections (internal/process/task_queues.js:97:5)

I have this problem too((( Dimension in a Docker container via the matrix-docker-ansible-deploy playbook on an CentOS 7 VM

TheTimeWalker commented 3 years ago

For anybody still following this issue: Can you try this with the latest version again? There was an update for the Telegraf dependency so maybe this could fix it for some people. Note that it still only works for non-animated sticker packs.

And0r- commented 3 years ago

nope I have tested https://t.me/addstickers/Blahaj with the current matrix-docker-ansible-deploy on ubuntu 20.04

And get still a 404 on logs

And0r- commented 3 years ago

Woops found the problem. it will need a Telegram token to download the stickers...

With the matrix-docker-ansible-deploy you have to add

matrix_dimension_configuration_extension_yaml: |
  # Your custom YAML configuration for Dimension goes here.
  # This configuration extends the default starting configuration (`matrix_dimension_configuration_yaml`).
  #
  # You can override individual variables from the default configuration, or introduce new ones.
  #
  # If you need something more special, you can take full control by
  # completely redefining `matrix_dimension_configuration_yaml`.
  #
  # Example configuration extension follows:
  # telegram:
  #   botToken: "YourTokenHere"

to your config and set a valid Telegram Token :)

Perhaps dimensions should check is there no token (or when possible is there a valid token) and raise a better error then 404^^

And0r- commented 3 years ago

The "Blahaj" is now working but there are a lot not animated packs that not works. e.g. https://t.me/addstickers/PEPEtop Perhaps there are to many stickers.

On packs with a lot not animated stickers and one animated, the import will abort and delete all not animated pictures.

I also found the Jan 19 16:30:15 synapse matrix-dimension[1483]: Error: Failed to locate Dimension metadata problem. this happen wehen you click in the chat self on stickers and try to download the pack on the "Your Sticker Packs" page.

(to add a sticker pack you have to use "edit widget/bot" -> config -> stickers as a dimension admin)

mx4k commented 3 years ago

@And0r- so it has to be a token for the HTTP API for a bot I have created?

Because, that doesn't work for me. I'm getting a 500 error.

image

{"message":"Error importing stickers","dim_errcode":"M_UNKNOWN","error":"Error importing stickers","errcode":"M_UNKNOWN"}

Philantrop commented 2 years ago

Importing stickers works fine for me after disabling rate-limiting in Matrix-Media-Repo (mmr) which I'm using. Even without mmr in the picture I can very well imagine rate-limiting playing a role in this.

WowSuchRicky commented 1 year ago

I ran into this as well and (mostly?) fixed it by doing what @And0r- suggested above and adding a Telegram bot token. You can message @Botfather on telegram (per their docs) and get your key that way.

When I say mostly I mean that there are still quirks here and there or certain sticker downloads may time out, but I do not see the 404 nor the "cannot modify a request after it is sent" errors anymore.