Services keep restarting

[odidev]

Hi Team,

I am working on the AWS x64 instance. I followed the documents here.

After executing tye run, I could see the services running on localhost:8000. However, services keep restarting, as shown below:

Documents suggest that it must be ssl certification issue. However, I could not find ssl issue in the logs on tye dashboard.

Also, I ran the angular application using yarn start. But, the application shows error on the GUI, as below:

Can you please provide pointers on the same.

[gterdem]

You need to check application logs under one of the restarting application Logs/logs.txt. It is probably because of the localhost.pfx file is not generated where it is used by tye for kestrel: https://github.com/abpframework/eShopOnAbp/blob/f1c51a2a2777d8784868f13fb0d0646a0f52b42f/tye.yaml#L18-L19

If that is the case, you can run create-certificate.ps1 that creates the SSL certificate: https://github.com/abpframework/eShopOnAbp/blob/f1c51a2a2777d8784868f13fb0d0646a0f52b42f/etc/dev-cert/create-certificate.ps1#L1

Or you can use run-tye.ps1 which automatically generates the certificates and run the applications.

[odidev]

create-certificate.ps1 and run-tye.ps1 are powershell formats, hence cannot be executed directly on the Linux/AMD64 AWS instance.

However, I tried creating certificate following the below command:

$ dotnet dev-certs https -v -ep localhost.pfx -p 8b6039b6-c67a-448b-977b-0ce6d3fcfd49 –t 

Trusting the HTTPS development certificate was requested. Trusting the certificate on Linux distributions automatically is not supported. For instructions on how to manually trust the certificate on your Linux distribution, go to https://aka.ms/dev-certs-trust 

[7] An error occurred while listing the certificates: System.Security.Cryptography.CryptographicException: Unix LocalMachine X509Store is limited to the Root and CertificateAuthority stores. 
---> System.PlatformNotSupportedException: Unix LocalMachine X509Store is limited to the Root and CertificateAuthority stores. 
   --- End of inner exception stack trace --- 
   at System.Security.Cryptography.X509Certificates.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags) 
   at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags) 
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.ListCertificates(StoreName storeName, StoreLocation location, Boolean isValid, Boolean requireExportable) 

A valid HTTPS certificate is already present. 

Looking at the error logs above, I am unable to say if certificate is already present or are there any errors.

And the services kept restarting on the tye dashboard while executing tye run.

[gterdem]

You can also use OpenSSL (or any similar tool) to generate self-signed certificate for linux something like:

openssl pkcs12 -export -out localhost.pfx -inkey privateKey.key -in localhost.crt -certfile localhost.crt

It checks the .pfx file located under the etc/dev-cert folder. Delete the pfx file and re-create it if you had problems during it's creation.

[odidev]

Thank you for the response.

I installed powershell on the ubuntu system and executed run-tye.ps1. All services started without restarting, except “keycloak-seeder”, which started and exited with exit code 0, and “catalog-service”, which exited with exit code 1. The same behaviour is seen on both the AMD64 and ARM64 platforms. Kindly find the tye dashboard below:

Log.txt for “catalog-service” shows the below logs:

2023-06-08 05:37:36.342 +00:00 [INF] Lock is acquired for db migration and seeding on database named: CatalogService... 
2023-06-08 05:37:47.523 +00:00 [INF] Initialized all ABP modules. 
2023-06-08 05:37:48.158 +00:00 [WRN] Overriding address(es) '[https://*:44354](https://%2A:44354/)'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead. 
2023-06-08 05:37:48.202 +00:00 [INF] Unable to bind to http://localhost:81/ on the IPv4 loopback interface: 'Permission denied'. 
2023-06-08 05:37:48.208 +00:00 [INF] Unable to bind to http://localhost:81/ on the IPv6 loopback interface: 'Permission denied'. 
2023-06-08 05:37:48.309 +00:00 [FTL] EShopOnAbp.CatalogService.HttpApi.Host terminated unexpectedly! 
System.IO.IOException: Failed to bind to address http://localhost:81/. 
---> System.AggregateException: One or more errors occurred. (Permission denied) (Permission denied) 
---> System.Net.Sockets.SocketException (13): Permission denied 
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress) 
   at System.Net.Sockets.Socket.Bind(EndPoint localEP) 
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint) 
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind() 
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass30_0`1.<<StartAsync>g__OnBind|0>d.MoveNext() 

May I know, is this the expected behaviour?

[gterdem]

Catalog service is using

• localhost:5000 for HTTP endpoint
• localhost:44354 for HTTPS endpoint
• localhost:81 for gRPC endpoint

https://github.com/abpframework/eShopOnAbp/blob/main/services/catalog/src/EShopOnAbp.CatalogService.HttpApi.Host/appsettings.Development.json

Your problem seems like kestrel is failing to bind the address. As I have found in this SO answer, you may try to run it with the admin privillages since it is about permissions.

[odidev]

Thank you for the response.

I worked under the root and now all the services are in the running state, as below:

Except the keycloak-seeder service, which isn’t failing, but seems like it did its required job and then exited with code 0.

May I know, is this the expected behaviour?

Also, proceeding further, I ran the angular app using yarn start.

Build at: 2023-06-12T07:35:11.053Z - Hash: 2dfcf5263d7aeed8 - Time: 47096ms 
** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ ** 

✔ Compiled successfully. 
✔ Browser application bundle generation complete. 
29 unchanged chunks 
Build at: 2023-06-12T07:35:14.571Z - Hash: 2dfcf5263d7aeed8 - Time: 2398ms 
✔ Compiled successfully. 

Application compiled successfully on the terminal, but nothing is accessible on the localhost:4200, as can be seen below:

Since it is the Angular Live Development Server listening on localhost:4200, is the above output justified?

[gterdem]

Except the keycloak-seeder service, which isn’t failing, but seems like it did its required job and then exited with code 0.

Keycloak-seeder is a hosted service, if you check its logs that it should seed the initial data successfully. It is normal to exit with code 0. But we may think about removing this dataseeder and move this responsibility to IdentityService if it seems confusing.

Application compiled successfully on the terminal, but nothing is accessible on the localhost:4200, as can be seen below:

localhost:44372 is the WebGateway, you can check the logs of the gateway to see where it makes a request and returns this error. It makes the first request to AdministrationService, so probably you may want to check the logs of the AdministrationService as well and share them so that we can see what the exact problem is.

[odidev]

I checked both Webgateway and Administration service logs but couldn’t find anything suspicious there.

However, in Administration service logs, there is a warning. Please find the same below:

2023-06-13 07:19:53.305 +00:00 [WRN] Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text='CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'', classId=0, methodId=0 
RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=320, text='CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'', classId=0, methodId=0 
   at RabbitMQ.Client.Framing.Impl.Connection.EnsureIsOpen() 
   at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.EnsureIsOpen() 
   at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.CreateModel() 
   at Volo.Abp.RabbitMQ.RabbitMqMessageConsumer.TryCreateChannelAsync() 

Kindly find the logs below:

1. WebGateway: logs_webgateway_eshoponabp.txt
2. Administration service: logs_administration_service_eshoponabp.txt

[odidev]

Hi Team, can you please provide some pointers in the above issue?

[gterdem]

It seems the RabbitMq was not ready or AdministrationService was not able to reach the RabbitMq. Then it seems application was shut down and restarted and then starts working normally as in the logs:

2023-06-13 07:25:36.666 +00:00 [INF]   - Volo.Abp.Identity.AbpIdentityHttpApiClientModule
2023-06-13 07:25:36.666 +00:00 [INF]     - Volo.Abp.Identity.AbpIdentityApplicationContractsModule
2023-06-13 07:25:36.666 +00:00 [INF]       - Volo.Abp.Users.AbpUsersAbstractionModule
2023-06-13 07:25:38.520 +00:00 [DBG] RabbitMQ Queue 'AbpBackgroundJobs.Volo.Abp.Emailing.BackgroundEmailSendingJobArgs' has 0 messages and 0 consumers.
2023-06-13 07:25:40.741 +00:00 [INF] Lock is acquired for db migration and seeding on database named: AdministrationService...
2023-06-13 07:25:54.870 +00:00 [INF] Lock is released for db migration and seeding on database named: AdministrationService...
2023-06-13 07:25:55.503 +00:00 [INF] Initialized all ABP modules.
2023-06-13 07:25:57.707 +00:00 [INF] Now listening on: https://[::]:44353
2023-06-13 07:25:57.717 +00:00 [INF] Application started. Press Ctrl+C to shut down.
2023-06-13 07:25:57.717 +00:00 [INF] Hosting environment: Development
2023-06-13 07:25:57.717 +00:00 [INF] Content root path: /home/ubuntu/eShopOnAbp/services/administration/src/EShopOnAbp.AdministrationService.HttpApi.Host

If there is still a problem, please create a new issue since the main issue seems resolved.