[Issue]: Invoice Payout issue

[siprapattanaik]

Describe the Issue

This is regarding HMC connector Invoice payout issue . We were on a DEFAULT program prior to our CANADA go live meaning all sellers were on DEFAULT program . After our CANADA go live , the existing sellers were moved to USA Hyperwallet Program . Our payout cycle is 15th for every month . What we observed today is that few of the invoices were not picked up by the HMC . On further analysis we found that we received errors for these invoices from HMC and the error detail is like this.

Something went wrong creating payment for invoice [230877] {exceptionMessage=You must provide a value for this field,error=CONSTRAINT_VIOLATIONS[[code=CONSTRAINT_VIOLATIONS,fieldName=programToken,message=You must provide a value for this field,relatedResources=]]}

The same set of sellers were moved to USA program after the go live . But why only a subset of sellers are experiencing these issues ? We would need more information as to why few sellers have issues and others do not have . Just for ex- both seller 232 Inc (2020) and seller Aosom(2094) were both under DEFAULT program prior to CANADA go live . After the go live both have program token as USA . However one seller 232 Inc (2020) had issue and did not get picked up whereas another seller Aosom(2094) did absolutely fine.

Environment

Live/Production

Version

4.13.1

Expected Behavior

No response

Actual Behavior

No response

Steps to Reproduce

No response

Pre-conditions

No response

Relevant log output

No response

[siprapattanaik]

[siprapattanaik]

We just grabbed the logs and seeing that there is no program token in the request .In addition to that there were successful payment calls on the same day as well where we see the program token...Both these are attached for reference

[javierpoloe2y]

Hi @siprapattanaik,

Thanks for reaching out.

Let me start by trying to sum up a the context of your reported issue:

• Up until now you have only been onboarding US sellers and been using a single program structure in HW. It means that all those existing sellers had, within the Mirakl shop view, their programToken as DEFAULT.
• Recently you have moved to a multi program structure in HW in order to cover US and CA. By doing that, all existing US sellers programToken values in Mirakl have normally been changed by you from DEFAULT to US.
• Newly onboarded CA sellers should have their programToken value in Mirakl set to CA.
• It seems that when executing the last invoiceExtractJob on the 15th of August some of your US sellers haven’t been paid because there wasn’t any programToken value sent over to HW.

We have several questions that would help us to investigate the issue:

1. I presume that the last successful execution of the InvoiceExtractJob was on the 15th of July. What HMC version were you using at that time?
2. For CA sellers, have they all been paid successfully? If any of those failed, could you please take one shop as example and provide the screenshot of the Mirakl shop view showing the paymentToken value?
3. For US sellers, it seems that some have been paid and others haven’t. Let split those US sellers in two categories:
4. Migrated US sellers who have been paid in the past (which means they were onboarded before the 15th of July on the single program). Question: Out of those migrated US sellers, how many have been successfully paid vs how many failed? If any of those failed, could you please take one shop as example and provide the screenshot of the Mirakl shop view showing the paymentToken value?
5. Fresh US sellers who have been onboarded after the migration to the multi program structure and were going to be paid this month. Question: Out of those fresh US sellers, how many have been successfully paid vs how many failed? If any of those failed, could you please take one shop as example and provide the screenshot of the Mirakl shop view showing the paymentToken value?
6. Before making the change to start using multiple programs, did you delete the database files and restarted the connector?

[siprapattanaik]

Hi @javierpoloe2y , Please find my replied inline. Just to give you little more context ,we had reached out to Mirakl support as well.They had asked us to resave the seller profile and try calling S20 API which we did locally and can see that shop id is retrieving the HW program as USA and then they asked us to reach to out to connector support.

1. I presume that the last successful execution of the InvoiceExtractJob was on the 15th of July. What HMC version were you using at that time?

Sipra(BBB) : Yes that's right. The HMC version is 4.13.1

2. For CA sellers, have they all been paid successfully? If any of those failed, could you please take one shop as example and provide the screenshot of the Mirakl shop view showing the paymentToken value?

Sipra(BBB) : There was only 1 CA seller which got paid successfully.

3. For US sellers, it seems that some have been paid and others haven’t. Let split those US sellers in two categories:

Sipra(BBB) : Yes that is correct .We had 143 USA sellers which are paid successfully and only 33 sellers have issues for the payouts.

4. Migrated US sellers who have been paid in the past (which means they were onboarded before the 15th of July on the single program). Question: Out of those migrated US sellers, how many have been successfully paid vs how many failed? If any of those failed, could you please take one shop as example and provide the screenshot of the Mirakl shop view showing the paymentToken value?

Sipra(BBB) : Out of the migrated sellers list , 119 sellers were paid successfully and there were failed ones as well(the count is 32). The screenshot is attached.

5. Fresh US sellers who have been onboarded after the migration to the multi program structure and were going to be paid this month. Question: Out of those fresh US sellers, how many have been successfully paid vs how many failed? If any of those failed, could you please take one shop as example and provide the screenshot of the Mirakl shop view showing the paymentToken value?

Sipra(BBB) :From this category we have 2 fresh USA seller failed in the payout.Screenshot is attached.

6. Before making the change to start using multiple programs, did you delete the database files and restarted the connector?

Sipra(BBB) : The connector was deployed when we moved to multiprogram and hence restarted . But we are not maintaining any database . So am really not sure which database files are you referring to .Could you please provide more context ?

[max6001]

Thanks for your response @siprapattanaik and great catch on hiding sensitive data on those screenshots ;) We're investigating on our side and will come back to you with potentially more question and/or suggested resolution.

Sorry again for the inconvenience.

[siprapattanaik]

Thanks @max6001

[siprapattanaik]

Hi @max6001 and @javierpoloe2y A quick update on the issue . When we spotted these issue on payment , we did a seller extract from Mirakl portal and saw that HW program is coming as DEFAULT even if in the Mirakl UI , we can see USA as the HW program.We reached out to Mirakl support . After Mirakl support asked us to resave the seller profile , we were able to see HW-program as USA . However HMC did not pick it up and we saw the same error of invalid program token . I just restarted the connector and it looks like it picked up the invoices . My question is that why only these 33 invoices had issues and not all . I have followed up with Mirakl support to see if there was any change on Mirakl side when they asked us to resave the seller profile

[javierpoloe2y]

Hi @siprapattanaik

Thanks for the update and glad to see the issue got resolved. You’ll find below a theory that could explain what happened.

Before going into the details I think it’s important to talk about this screenshot:

What you see is a piece of data we are keeping in a database cache when the HMC is running. We keep track of relations between shops, program tokens, bank account tokens, ... in order to speed up the HMC and also avoid hitting the rate limit on the Mirakl side. By default, we pull information such as the program from that cache.

Regarding the database is important to understand that although the connector doesn't need an external database, it uses an embedded database that stores the information in:

• a './data' subfolder, if the connector is launched as a java application
• inside the docker container, if the connector is launched with docker using the provided docker-compose file. This means that for example if the docker container is deleted the information stored in the database will also be deleted.

Also is important to note two things:

• what is stored in the cache and also what is retrieved from Mirakl are program names (like DEFAULT or USA)
• to resolve the programToken that is sent to Hyperwallet, the connector will search in its configuration for a property that maps a program name to a program token.

Now the following is what we think happened:

• You changed the connector configuration renaming DEFAULT program to USA and adding CA program.
• You restarted the connector (this erased all database files so the persistent cache was empty after restart).
• You changed the seller’s programToken value in Mirakl from DEFAULT to USA.
• The billing cycle generated the invoices.
• The connector retrieved the invoices:
  • The connector started by searching for the program values in the cache.
  • Because the cache was empty due to the restart of the HMC, the connector called Mirakl APIs to retrieve the program value for each shop and saved that value into the cache.
  • 32 sellers invoices weren’t able to be processed because Mirakl incorrectly returned DEFAULT as program value while the Mirakl interface was showing USA.

The following is what you did to fix the issue:

• After contacting Mirakl you re-saved the failing 32 shops to ensure that the program value was correctly updated from DEFAULT to USA.
• You run the invoice job, and the connector first tried to get the program value from the cache.
  • Because the cache had DEFAULT from the previous execution, the HMC kept that value and disregarded the fact it had been updated in Mirakl.
  • So the invoice processed failed again for those invoices.
• You then fully restarted the HMC which cleared up the cache.
• You re-executed the InvoiceExtractJob. Because the cache was empty it pulled the data from Mirakl and if Mirakl properly returned the values then it worked.

We will close this ticket but don’t hesitate to reach out if needed. Best,