search

shlinkio / shlink

The definitive self-hosted URL shortener
https://shlink.io
MIT License
3.32k stars 271 forks source link

When trying to see visits info on a tag with a lot of links even with few visits getting an error #781

Closed levdad closed 4 years ago

levdad commented 4 years ago

How Shlink is set-up

Summary

When trying to see visits info on a tag with a lot of links even with few visits getting an error

Current behavior

returning an error message "An error occurred while loading visits :("

Expected behavior

see visits data by tag

How to reproduce

having a tag with around 100,000 links and trying to get visits stats image

Logs

[2020-06-07T08:43:55.188044+00:00] [6db0649a-c9db-4af3-8b18-2e4d8fd649d3] Access.INFO - 62.219.134.128 - - "GET /rest/v2/health HTTP/1.1" 200 120 [2020-06-07T08:43:55.300241+00:00] [79d38464-3976-4c23-9026-94b32b880133] Shlink.ERROR - Shlinkio\Shlink\Rest\Exception\MercureException: This Shlink instance is not integrated with a mercure hub. in /etc/shlink/module/Rest/src/Exception/MercureException.php:21 Stack trace:

0 /etc/shlink/module/Rest/src/Action/MercureInfoAction.php(35): Shlinkio\Shlink\Rest\Exception\MercureException::mercureNotConfigured()

1 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/RequestHandlerMiddleware.php(54): Shlinkio\Shlink\Rest\Action\MercureInfoAction->handle(Object(Laminas\Diactoros\ServerRequest))

2 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Laminas\Stratigility\Middleware\RequestHandlerMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

3 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

4 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

5 /etc/shlink/vendor/mezzio/mezzio-router/src/Route.php(101): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

6 /etc/shlink/vendor/mezzio/mezzio-router/src/RouteResult.php(111): Mezzio\Router\Route->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

7 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/DispatchMiddleware.php(36): Mezzio\Router\RouteResult->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

8 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Mezzio\Router\Middleware\DispatchMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

9 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

10 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

11 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

12 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/PathMiddlewareDecorator.php(130): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

13 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(54): class@anonymous->handle(Object(Laminas\Diactoros\ServerRequest))

14 /etc/shlink/module/Rest/src/Middleware/AuthenticationMiddleware.php(44): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

15 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Shlinkio\Shlink\Rest\Middleware\AuthenticationMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

16 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

17 /etc/shlink/module/Rest/src/Middleware/BodyParserMiddleware.php(42): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

18 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Shlinkio\Shlink\Rest\Middleware\BodyParserMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

19 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

20 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/ImplicitOptionsMiddleware.php(71): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

21 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Mezzio\Router\Middleware\ImplicitOptionsMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

22 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

23 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

24 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/PathMiddlewareDecorator.php(68): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(class@anonymous))

25 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\Middleware\PathMiddlewareDecorator->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

26 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(54): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

27 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/ImplicitHeadMiddleware.php(85): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

28 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Mezzio\Router\Middleware\ImplicitHeadMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

29 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

30 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/RouteMiddleware.php(55): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

31 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Mezzio\Router\Middleware\RouteMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

32 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

33 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

34 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

35 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(54): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

36 /etc/shlink/vendor/shlinkio/shlink-common/src/Middleware/CloseDbConnectionMiddleware.php(27): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

37 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Shlinkio\Shlink\Common\Middleware\CloseDbConnectionMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

38 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

39 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

40 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

41 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/PathMiddlewareDecorator.php(130): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

42 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(54): class@anonymous->handle(Object(Laminas\Diactoros\ServerRequest))

43 /etc/shlink/vendor/mezzio/mezzio-problem-details/src/ProblemDetailsMiddleware.php(60): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

44 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Mezzio\ProblemDetails\ProblemDetailsMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

45 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

46 /etc/shlink/vendor/php-middleware/request-id/src/RequestIdMiddleware.php(32): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

47 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): PhpMiddleware\RequestId\RequestIdMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

48 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

49 /etc/shlink/module/Rest/src/Middleware/CrossDomainMiddleware.php(23): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

50 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Shlinkio\Shlink\Rest\Middleware\CrossDomainMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

51 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

52 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

53 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/PathMiddlewareDecorator.php(68): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(class@anonymous))

54 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\Middleware\PathMiddlewareDecorator->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

55 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(54): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

56 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/ErrorHandler.php(144): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

57 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Laminas\Stratigility\Middleware\ErrorHandler->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

58 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

59 /etc/shlink/vendor/mezzio/mezzio-helpers/src/ContentLengthMiddleware.php(32): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

60 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(47): Mezzio\Helper\ContentLengthMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

61 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

62 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

63 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))

64 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))

65 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(73): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\EmptyPipelineHandler))

66 /etc/shlink/vendor/mezzio/mezzio-swoole/src/SwooleRequestHandlerRunner.php(244): Laminas\Stratigility\MiddlewarePipe->handle(Object(Laminas\Diactoros\ServerRequest))

67 {main}

[2020-06-07T08:43:55.301705+00:00] [79d38464-3976-4c23-9026-94b32b880133] Access.ERROR - 62.219.134.128 - - "GET /rest/v2/mercure-info HTTP/1.1" 501 161

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267 [2020-06-07 08:43:55 *156.53] ERROR php_swoole_server_rshutdown (ERRNO 503): Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267 [2020-06-07T08:43:55.820070+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 53

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267 [2020-06-07 08:43:56 *157.62] ERROR php_swoole_server_rshutdown (ERRNO 503): Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267 [2020-06-07T08:43:56.466922+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 62

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267 [2020-06-07 08:43:57 *151.51] ERROR php_swoole_server_rshutdown (ERRNO 503): Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267 [2020-06-07T08:43:57.052366+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 51

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267 [2020-06-07 08:43:57 *158.53] ERROR php_swoole_server_rshutdown (ERRNO 503): Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267 [2020-06-07T08:43:57.706185+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 53

acelaya commented 4 years ago

That error you see in the logs is expected.

When you connect to a Shlink server from shlink-web-client, it does a one-time request to the GET /rest/v2/mercure-info endpoint, in order to know if the server supports the integration with mercure and it is configured.

If the endpoint returns a 501 response, then shlink-web-client knows it is not configured, ignoring it after that moment.

It should probably not be logged as an error though, as it's an expected behavior and therefore, it can be confusing.

So, my guess is that there's probably some other error causing the "An error occurred while loading visits :(", probably something related with loading the visits from the database, as it's very unlikely that the total amount of visits to 100k short URLs combined is only one.

Could you open the browsers console and check for XHR requests to endpoints ending with /visits? Some of those requests is probably failing. Click on that request and search for a x-request-id header in the response. Take the UUID returned there, and search for it in the logs.

With that, you can see all the logs generated during that request, and one of them should be the actual error causing that.

levdad commented 4 years ago

Thanks for your fast response.

It actually doesn't return an x-request-id . The error is probably something to do with memory Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267 [2020-06-07 08:43:55 *156.53] ERROR php_swoole_server_rshutdown (ERRNO 503): Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /etc/shlink/vendor/doctrine/lexer/lib/Doctrine/Common/Lexer/AbstractLexer.php on line 267

The reason why there are a lot of links and very few visits is because we are trying to use shlink as our short-link generator instead of the one we are using right now. So i imported all the links from the our current short link provider so that when we make the move to shlink all existing links will still work.

acelaya commented 4 years ago

The reason why there are a lot of links and very few visits is because we are trying to use shlink as our short-link generator instead of the one we are using right now. So i imported all the links from the our current short link provider so that when we make the move to shlink all existing links will still work.

Got it, that makes sense.

The error is probably something to do with memory.

Ok, this one makes more sense, and I think I know what could be causing it. I need to do some testing and once I have it fixed I will release a patch version.

Thanks for reporting it.

acelaya commented 4 years ago

I have just reproduced the issue. It was due to an attempt to simplify queries by getting all the IDs of the short URLs for selected tags, in order to avoid some joins on a later query.

It proved to be a performance improvement for the single-short-url visits use case, but in there I just had to fetch one ID, while in this use case the value can grow to, potentially, 100k like in this case.

So I have changed it to use the join in the case of visits for a tag.

I used this for the tests and it is fixed now:

image

I want to include another bugfix on this release, but after that, I will publish v2.2.2

levdad commented 4 years ago

Thank you very much!

acelaya commented 4 years ago

Hey @levdad. I have just released v2.2.2, which should fix this issue.

Let me know if something else comes up.

helmut72 commented 2 years ago

Get the same output on v3.0.3. Never used an older version.