microsoft / service-fabric

Service Fabric is a distributed systems platform for packaging, deploying, and managing stateless and stateful distributed applications and containers at large scale.
https://docs.microsoft.com/en-us/azure/service-fabric/
MIT License
3.03k stars 401 forks source link

Service Proxy problems with the latest version of service fabric runtime/sdk #702

Open Lemraj opened 6 years ago

Lemraj commented 6 years ago

Hello guys,

I have multiple service fabric services. Some of them have a listener name, others not. All services worked fine till version 6.3.162/3.2.162. When I updated the service fabric packages to the latest version (6.3.176/3.2.167), I am experiencing problems with the services without a specific listener name. The scenario is as follow: 1 - First I am calling a service with a specific listener name using ServiceProxy 2 - After that I do a call to the service without a listener name using ServiceProxy.

The first call is doing fine and no problems, but the second call (no listener name) throws this error:

NamedEndpoint 'NameOfTheListenerOfTheFirstServiceCall' not found in the address '{"Endpoints":{"":"ComputerX:30015+f8cd5780-ae3a-430d-a5d8-caf30c645af0-131777691239185654-1d49c578-b8b6-4cc1-88fc-53030cfee8b3"}}' for partition 'f8cd5780-ae3a-430d-a5d8-caf30c645af0'

It took me some time to discover that the proxy object of the second call has the listener name of the first service call!!! The listener name of the second call should be null (empty), because I specifically fill null as a parameter.

I did a rollback to the previous version and all worked fine, so it must be something in the latest version causing this problem.

As a workaround, I managed to solve this problem using the ServiceProxyFactory. But still I want to keep using the ServiceProxy in stead of ServiceProxyFactory.

Is anyone experiencing this problem?

Thank you in advance.

rwwilden commented 6 years ago

Seeing the same issue here. Seems something is cached or singleton where it should not be.

The work-around works but since the docs state that creating a ServiceProxyFactory is expensive, this is not desirable.

Lemraj commented 6 years ago

@masnider, any progress on this issue? Thanks!

aresundnes commented 5 years ago

Having the same problem here..!

Lemraj commented 5 years ago

@masnider, @BharatNarasimman, @VipulM-MSFT, @suchiagicha:

Any progress please? it has been almost 3 months since I added this issue. :(

suchiagicha commented 5 years ago

@Lemraj Yes .This has been fixed in 3.2.176. You can use any version after that or same.

Lemraj commented 5 years ago

@suchiagicha, @masnider, @BharatNarasimman I still have the same problem! I am using version 6.4.654.9590.

Can someone of you please tell me (us) why this still occurs? I have to use the work-around I mentioned before, but according to the docs this is an expensive operation and not desirable.

suchiagicha commented 5 years ago

@Lemraj

Are you seeing the same exception? Also in your previous issue , https://github.com/Azure/service-fabric-issues/issues/1221 I saw you are using same proxyfactory(default) to talk to V1 and V2_1 stack? If thats the case , it wont work. We determine the stack version at proxyfactory level or first Serviceproxy call. You need to create different proxyfactory for different remoting stack.

Lemraj commented 5 years ago

@suchiagicha microsoft/service-fabric-issues#1221 is not added by me! I am having the same issue mentioned in this issue microsoft/service-fabric-issues#1265!

I am using the V2 stack.

suchiagicha commented 5 years ago

@Lemraj

Could you please give us the sample. Want to add one more thing .Seeing your error , looks like your service is publishing one endpoint with "" listenerName. Are you sure you have use listener Name while registering your listener in service code?

NamedEndpoint 'NameOfTheListenerOfTheFirstServiceCall' not found in the address '{"Endpoints":{"":"ComputerX:30015+f8cd5780-ae3a-430d-a5d8-caf30c645af0-131777691239185654-1d49c578-b8b6-4cc1-88fc-53030cfee8b3"}}' for partition 'f8cd5780-ae3a-430d-a5d8-caf30c645af0'

Lemraj commented 5 years ago

@suchiagicha

This is the error I get:

NamedEndpoint 'V2Listener' not found in the address '{"Endpoints":{"":"DESKTOP-HOME:32072+09a60054-a332-4bdf-ad5f-10369daefd1c-132012929766469144-de11a972-bcc7-4176-b52d-087998e4ce46"}}' for partition '09a60054-a332-4bdf-ad5f-10369daefd1c'

This code doesn't work: var uriBuilder = new ServiceUriBuilder("CoC"); var proxy = ServiceProxy.Create<ICoC>(uriBuilder.ToUri()); var result = await proxy.GetBasicInfoAsync(number, cancellationToken);

QuickWatch of proxy object: image

ServiceUriBuilder class used can be found here

The scenario that works using ServiceProxyFactory : Uri _cocUri = new Uri("fabric:/SF/CoC"); var proxy = ProxyFactory.ServiceProxyFactory.CreateServiceProxy<ICoC>(_cocUri); var result = await proxy.GetBasicInfoAsync(number, cancellationToken);

QuickWatch of proxy object: image

suchiagicha commented 5 years ago

@Lemraj How are you using V2 stack? Are you using assembly attribute? Could you share remoting interface? How are you registering listener?

Complete sample would be helpful in order to avoid all these questions. :)

Lemraj commented 5 years ago

@suchiagicha

First of all, thank you for your support! This are the files I think you need to see:

ServiceManifest.xml image

ICoC.cs interface image

CoC.cs service implementation image

Program.cs image

suchiagicha commented 5 years ago

@lemraj if you are using attribuites in your ICOC interface , you should not register using listener. You should use below code to register listener. Basically we have 2 ways to specify remoting stack. One using attribute , another explicit class instantiation but you have mixed both. https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-communication-remoting#use-the-remoting-v2-interface-compatible-stack Change your CreateServiceInstanceListeners code to below one and see if it helps. Also there is known issues with V2 serialization , wud recommend to use V2_1 stack.

protected override IEnumerable CreateServiceInstanceListeners() { return this.CreateServiceRemotingInstanceListeners(); }

Lemraj commented 5 years ago

@suchiagicha

I tried your suggestion and it did work (but not always) on my local machine (cluster). I deployed the application to the develop environment and I constantly get the same error again.

So for now I had to roll back again to the workaround I already used.

suchiagicha commented 5 years ago

@Lemraj I am not sure then whats wrong. We are not able to repro this issue on our side. If you an provide us your running repro project, we can run on our machines and then debug it.

Lemraj commented 5 years ago

@suchiagicha

Thank you for your support! Due to company policy, I can not share the whole repro.

I think the problem has to do with the combination of services I am using. Some of them have a listener and others not.

Calling the first service with listener is fine! But the proxy object of the second call to the service without a listener, has the listener name of the first service call!!! The listener name of the second call should be null (empty), because I specifically fill null as a parameter.

I will try to reproduce it in a separate project and will come back to you if this is still the case.

suchiagicha commented 5 years ago

@lemraj What is the reason for you to use different proxy with listenerName ? Is to issue calls for remoting V1 and V2?