Invalid Billing Agreemend ID. A valid PayPal Billing Agreement ID must begin with "B-"

[kmg-yrabelo]

Hello Prospress,

We have been gathering the data for all of our customers in the past week and while trying to test the import script with a smaller subset of our data, we encountered the following error:

Invalid Billing Agreemend ID. A valid PayPal Billing Agreement ID must begin with "B-" http://screencast.com/t/EAtFYgojpoQF

A quick search on our database for the value of _paypal_subscription_id rendered the following results:

Starts with B: - 660 records Starts with I: 7180 records Everything Else: 1822 records

Does this mean that only those subscriptions where _paypal_subscription_id starts with a B will be imported and the rest can't be imported successfully?

[mattallan]

Hey @kmg-yrabelo!

Thanks for bringing this to our attention! We're going to have to change a few things with the importer code to allow PayPal IDs that start with I- (PayPal Standard IDs) to be imported.

Given the many PayPal Standard limitations, migrating these subscriptions have a lot of room to fail and cause issues when it comes time for them to renew (this is why we don't allow you to import these in the current master branch).

That said we're going to try include a patch that will allow you to import these, and we'll provide you with some tips to avoid issues.

We'll keep you posted! It shouldn't take too long :) Thanks

---

Read at your own risk.

If you want to continue through the importing process with the current master branch, you can setup your csv by moving all the PayPal subscription IDs that start with I- into a separate column _paypal_subscription_id and map it to custom_post_meta and then make sure you have the payment_method column set to paypal. Example: https://cloudup.com/cpXDdQ4qQQS

When you do the import, you'll see a warning like: No payment meta was set for you paypal subscription, but you can ignore this because the subscription will still be updated with the I- paypal ID.

When importing these I- paypal subscriptions, it's crucial that you make sure the IPN URL on the PayPal account is pointing to stores current URL (otherwise customers will be charged and no renewal order will be created).

This is a hack and is only possible because your store has PayPal RT enabled, but we're going to make this a little easier to manage and provide more information to users.

Patch incoming shortly :)

[kmg-yrabelo]

Thank you, @mattallan . Looking forward to your patch so I can give it another try.

[thenbrent]

@kmg-yrabelo I just added some notes on https://github.com/Prospress/woocommerce-subscriptions-importer/pull/125#issuecomment-231810904 to explain why that PR won't be merged, we'll try to come up with another solution in the next couple of days for you.

[mattallan]

@kmg-yrabelo unfortunately we won't be able to fully support your PP Standard subscriptions (the ones starting with I-) in the importer due to the limitations that I linked to in: https://github.com/Prospress/woocommerce-subscriptions-import-export/issues/124#issuecomment-230999248, specifically this one: https://docs.woocommerce.com/document/subscriptions/store-manager-guide/#paypal-ipn-limitation

Can just confirm that the store is moving all their subscriptions to a different URL? Sorry for the delay.

[kmg-yrabelo]

@mattallan , @thenbrent ,

Thank you for your help on this issue. We are not going to be wiping out every order, subscription, and memberships. The users and user_meta table are going to be untouched. The URL is not going to change, we are going to be upgrading on a staging site at WPEngine and then switch databases and make sure everything points to the main kelbyone.com domain.

We are going to use the route that @mattallan suggested above and let it throw the warning message.

[thenbrent]

Can just confirm that the store is moving all their subscriptions to a different URL?

The URL is not going to change

Unfortunately, it still won't work even if the base URL is the same.

WooCommerce only listens to IPN messages on its own IPN URL, e.g. http://example.com/?wc-api=WC_Gateway_Paypal or http://example.com/wc-api/WC_Gateway_Paypal. So unless the IPN messages come through to that URL, neither WooCommerce nor Subscriptions will be able to process them to handle renewal payments and keep subscription statuses in sync (e.g. if the subscription is suspended at PayPal, it also gets suspended in WooCommerce).

As @mattallan mentioned, although we have previously received information from PayPal reps that the IPN URL can be changed on an account and new IPNs will be sent to that URL, in practice we can not confirm this or reproduce this behaviour in the sandbox. Instead, we are seeing the IPN URL set on the subscription at the time it was created used for all future IPN messages.

[alanpace]

Can you please confirm that it's still impossible to import PayPal standard subscriptions that weren't originally created with the WooCommerce IPN URL? I need to import about 900 subscriptions with ID's starting with I-, originally created from a PayPal custom button.

[thenbrent]

@alanpace yes unfortunately this is still the case.

PayPal only send IPN messages to the URL used when the subscription was created, so there's nothing we can do.

You could possibly write a custom script at the old URL to validate IPN messages and notify the store of them with more custom code to process the payment. But that is extremely complicated to write, very difficult to test and ultimately very flaky even after you get it working.

[alanpace]

I think I've thought of a relatively simple solution that will solve the issue with the IPN URL. This article seems to confirm that it will work, but I haven't tested it yet: http://www.idowebdesign.ca/server-stuff/redirect-paypal-ipn-with-htaccess/

The site I'm working on is switching from ZenCart to WooCommerce. At the time the subscriptions to be imported were created, the default IPN was set to point to the ZenCart IPN. However, since ZenCart doesn't support subscriptions, the subscriptions were created from PayPal custom buttons. When a subscription is created using a PayPal button, is the default IPN URL that was set in the merchant's PayPal account associated with the subscription? Is there any way to identify what IPN URL is associated with a subscription?

[alanpace]

I answered my own question. From the PayPal profile page for the Merchant, choose "My Selling Tools", then select update next to "Instant Payment Notifications", and click the link on that page to the "IPN History Page".

[DaveHamilton]

We've been dealing with this migration of PayPal Standard Transactions (the "I-" type), too, albeit in a manual sense, and we've learned a few things.

First, the IPN URL does change for all transactions when you update it on PayPal's side, or at least it did for us. We previously used Open Gateway and when we updated the IPN URL on PayPal, all those old OG transactions immediately began coming into the WooCommerce IPN URL, and they show up in the WC logs.

So that's good.

Unfortunately, though, even though we have put the correct/matching "I-" PayPal Subscription ID into Woo, it seems that Woo doesn't actually match on that. It only uses that for outbound requests. Woo is able to suspend and reactivate these existing (pre-Woo) PayPal Subscription IDs, but inbound requests from PayPal with those IDs seem to do nothing other than result in Woo saying "Received valid response from PayPal."

My assumption (please correct me if I'm wrong, of course) is that the "custom" field is the one that Woo looks at for action:

[custom] => {"order_id":6869,"order_key":"wc_order_xxxxxxxxxxxx","subscription_id":6870,"subscription_key":"wc_order_xxxxxxxxxxxx"}

We're so close here. Everything's happening correctly, Woo just doesn't know that it's supposed to spring into action to automatically create the order and log the payment.

And we want it to. ;)

So the question is: has someone yet developed a way to have Woo either (a) trigger off of these inbound IPN requests, perhaps keying in on the PayPal Subscription ID, or (b) query PayPal and have the query script then trigger Woo into doing this?

I really don't want to ask my customers to cancel and resubscribe... that's bad business. And all the data is here. We just need it wired up. Anyone already solve this problem?

[thenbrent]

My assumption (please correct me if I'm wrong, of course) is that the "custom" field is the one that Woo looks at for action:

That's wrong. :)

Subscriptions attempts to match the IPN on a variety of data. You can see this in WCS_PayPal_Standard_IPN_Handler::get_order_id_and_key().

Most importantly for this case, if the IPN payload has a subscr_id value, Subscriptions will search for a subscription post with a matching '_paypal_subscription_id' post meta value.

A summary of the query it uses is:

            $posts = get_posts( array(
                'numberposts'      => 1,
                'orderby'          => 'ID',
                'order'            => 'ASC',
                'meta_key'         => '_paypal_subscription_id',
                'meta_value'       => $subscr_id_value_from_paypal_ipn_payload,
                'post_type'        => 'shop_subscription',
                'post_status'      => 'any',
                'suppress_filters' => true,
            ) );

It's only when it can't find a subscription post with a matching '_paypal_subscription_id' post meta value that it then attempts to use the custom data in the IPN payload.

It's possible there is a bug in the WCS_PayPal_Standard_IPN_Handler::get_order_id_and_key() code, it's extremely hard to test, but the more likely issue is that the '_paypal_subscription_id' post meta value has not been set on the subscription. Make sure that meta value is set and Subscriptions should identify the matching subscription for IPNs without WooCommerce's custom payload.

First, the IPN URL does change for all transactions when you update it on PayPal's side, or at least it did for us.

That's good to know, thanks for sharing that info! I've received mixed reports from PayPal reps about this (some say it does update, some say it doesn't). IIRC, in the few times I've tested this, I've also received mixed results from different sandbox accounts.

If we can find a way to make it work, we can at least document all the manual steps required to help others. I'm not sure it will be "officially supported" by the importer given how much needs to happen outside of the code. It's just too error prone.

[DaveHamilton]

Subscriptions attempts to match the IPN on a variety of data. You can see this in WCS_PayPal_Standard_IPN_Handler::get_order_id_and_key().

Most importantly for this case, if the IPN payload has a subscr_id value, Subscriptions will search for a subscription post with a matching '_paypal_subscription_id' post meta value.

Ahh! Very helpful. Thank you.

Unfortunately, the inbound IPN from PayPal includes that as recurring_payment_id, not subscr_id, so that at least explains why it's failing.

We definitely have the '_paypal_subscription_id' value set. We had to do it manually in the database, but it's there. And the fact that we're able to use Woo to suspend/reactivate those PayPal subscriptions seems to indicate that it's in the right place.

I guess the question is this: is there any way to have Subscriptions look at recurring_payment_id on the inbound request? Because there's probably no way to have PayPal change that from recurring_payment_id to subscr_id – but, again, please do correct me if I'm wrong. ;)

[thenbrent]

I guess the question is this: is there any way to have Subscriptions look at recurring_payment_id on the inbound request?

It should be doing that already. It falls back to use the 'recurring_payment_id' IPN value when 'subscr_id' is not available. Screenshot of the code: https://cloudup.com/cLKXoP0zzZi

And the fact that we're able to use Woo to suspend/reactivate those PayPal subscriptions seems to indicate that it's in the right place.

Yes that does indicate it's in the right place...

[DaveHamilton]

Ok, that code helps, thank you! It's looking for that data on the order, not on the subscription. The problem is that we had manually done Actions > Create Pending Renewal Order for one of these folks, and we just confirmed that it does have this field on there, but it still didn't work.

One difference we noticed in comparing the two is that a subscription started in Woo seems to have one order marked as "Parent Order" whereas one we manually added just has "Renewal Order" attached to it.

It seems like we're missing something simple in our manual import process here... perhaps?

[phylaxis]

@thenbrent I just went through that WCS_PayPal_Standard_IPN_Handler::get_order_id_and_key() function and compared it to the order records we have in our database. Looking at everything to me it seems like this function is not the problem. As far as I can tell that function should be finding the $order_id and $order_key values it's looking for. To me the issue looks like it would be happening after this function.

[thenbrent]

To me the issue looks like it would be happening after this function.

@phylaxis that's possible. Are you seeing any Subscription IPN Error: log entries to explain the IPN handling existing early?

Entries like:

Subscription IPN Error: Subscription Key does not match invoice.

[phylaxis]

There are no messages like that related to the orders we're looking at. No.

[DaveHamilton]

Ok, thanks for this, @thenbrent. You've helped us quite a bit. We've done some testing and some digging and here's the issue: it's never getting to the point where it tests for 'recurring_payment_id'.

In class-wcs-paypal.php at line ~308 it's looking for a matching txn_type and txn_type=recurring_payment is what we're getting via the IPN request. That's not in the list of accepted $transaction_types at line 25 of class-wcs-paypal-standard-ipn-handler.php, so... that's where it all ends.

And now, of course, the question: how do we get it to accept txn_type=recurring_payment and process the inbound request? :)

(edit: happy to send you a raw IPN request from one of these folks so you can see the inbound data for yourself, too)

[thenbrent]

@DaveHamilton ah OK that makes sense. That's a much bigger, more painful issue. That means the IPN is for a Subscription using PayPal Express Checkout Recurring Payments, not PayPal Standard Subscriptions, which is what Subscriptions supports. Those are two separate PayPal products/APIs.

how do we get it to accept txn_type=recurring_payment and process the inbound request? :)

You'll need an IPN handler for PayPal Express Checkout Recurring Payments. The IPN payloads for those are different to the payloads for PayPal Standard Subscriptions.

The really bad news is that I don't know of one for WooCommerce. None of the PayPal Express Checkout extensions I know of (i.e. the ones released through WooCommerce.com) include Subscriptions support (for various reasons). There is code in the PayPal Digital Goods extension that would be a good starting point because PayPal Digital Goods uses the Express Checkout APIs, but it's definitely not written to handle imported subscriptions so there's likely a number of gotchas along the way.

[phylaxis]

@thenbrent Sounds like really bad news for us. So I better understand this… even if we found an IPN handler for PayPal Express Checkout Recurring Payments that worked with WooCommerce how would we tie that in with WooSubscriptions? Or is that not possible? What we are trying to to is migrate existing subscribers from an entirely different platform on to WooCommerce with WooSubscriptions. So at this point do you think that is not possible?

[thenbrent]

even if we found an IPN handler for PayPal Express Checkout Recurring Payments that worked with WooCommerce how would we tie that in with WooSubscriptions? Or is that not possible?

It's possible, it's just potentially a lot of work.

The best case is that you can find a PayPal Express Checkout plugin for WooCommerce that already supports WooCommerce Subscriptions. Then you should only need to make sure it can handle IPNs for PayPal Recurring Payments created with a different platform. This may happen out-of-the-box, or it may require just a little work to decouple the IPN handling from WC specific IPN data, like custom (in line with the conversation above).

The worst case is that you need to build one from scratch.

I just checked and the PayPal Digital Goods extension's IPN handler does have code to handle IPNs for recurring payments from other systems, so that's probably a good start if you can't find a PayPal Express Checkout specific WooCommerce plugin that supports WooCommerce Subscriptions.

I'm not sure of all the differences between PayPal Digital Goods and PayPal Express Checkout, I just know that Digital Goods was built on the Express Checkout APIs, so it's very similar. That said, PayPal has discontinued their Digital Goods product, so it's possible they've continued adding features to Express Checkout that aren't included in Digital Goods.

[DaveHamilton]

Thanks, @thenbrent. Seems we've got some research to do. Thank you VERY much for the assist here. I do have one lingering question that may betray my lack of understanding (again!), but, hey, why not go for it all?

Right now it's obvious to us that whatever we set as the IPN destination URL over at PayPal will be the IPN destination for everything on our account, old and new alike. If we use/adapt something like the PayPal Digital Goods extension would that require one IPN URL for the old stuff to point to that and one IPN URL for the new stuff? Or would hooking it into WooCommerce Subscriptions be all that's required?

Obviously if we need two different IPN URLs for the two different types of transactions, we're out of luck and will need to explore a different methodology, perhaps adopting something similar to what Open Gateway does. That seems to ignore the inbound IPN stuff altogether and instead has a daily cron script that goes out and fetches the data from PayPal in a pull request.

[thenbrent]

hooking it into WooCommerce Subscriptions be all that's required?

Hooking into WooCommerce should be all that's required. As long as you can get the IPN to come to the WooCommerce site URL, then your can have (custom) code to process it and connect to Subscriptions/WooCommerce etc.

[DaveHamilton]

Oh! You rock. That makes perfect sense. And, based upon our testing, there are only two things that we think need to change, they are:

Change field name: recurring_payment_id > subscr_id (keeping contents the same) Change field contents of txn_type from "recurring_payment" -> "subscr_payment"

And, with some testing, we did this. And it works. Kind of. The payment processes and gets logged to the subscription (w00t!), but no renewal order is created and we also get an error:

Fatal error: Call to a member function payment_complete() on a non-object in [...]plugins/woocommerce-subscriptions/includes/gateways/paypal/includes/class-wcs-paypal-standard-ipn-handler.php on line 318

That line number may not be exactly right due to some of the testing we've done. Our assumption is that the issue is no prior order exists to match the subscr_id – only a subscription – and that's causing WooCommerce Subscriptions to fail because it can't create the renewal order.

So my first question is: what should our import process be for a PayPal subscription? Do we need to create an Order? For an Authorize.net subscription we can just create the Subscription and then the first Order is created when the payment is processed. But for PayPal this seems not to work.

And after we answer that first question and do it "right" we'll then likely have a question about where to hang the code to do those aforementioned field mappings. ;) But our current question is about the import/manual subscription creation process.

Thanks, @thenbrent.

[DaveHamilton]

Our inbound IPNs all worked based on the above logic and assumptions. Seems with PayPal we need to have not only a subscription created in WooCommerce, but we also need to have a previously-paid order attached to that subscription, otherwise the inbound IPN can't find the subscription properly.

That leaves us with the lingering question about our manual import process: is it correct to create the subscription, the order AND then go into the wp_postmeta table and set _paypal_first_ipn_ignored_for_pdt to true so that WooCommerce Subscriptions doesn't ignore the first inbound payment?

[thenbrent]

@DaveHamilton yeah unfortunately I think you'll need to do something like that.

You can create additional code to add the first order during or after the import with the wcs_create_renewal_order() function, or do it manually via the Edit Subscription screen for each one.

You are also right that '_paypal_first_ipn_ignored_for_pdt' will need to be set. I think that will be everything you'll need to do to make sure they are handled correctly from then on. But as you've already seen, the PayPal Standard IPN handling is awfully complex with lots of corner cases being guarded against.

[DaveHamilton]

@thenbrent thank you! This has been immensely helpful as we've sorted this all out. I do have a related question about where we should hang the code doing the custom mappings that we had to do for PayPal Express Checkouts to work, but I think it probably makes sense to open up a new ticket to do that, so we'll formulate all that into something coherent and put that up. Assuming you don't just tell me to continue the thread here, I'll certainly link to that ticket from here for anyone following the trail down the road. Thanks!

[DaveHamilton]

For those catching up here, we have had great success accepting inbound IPN requests for previously-created PayPal Express Checkout transactions by simply mapping thetxn_type from recurring_payment to subscr_payment. Treating anything that comes in as recurring_payment like it came in as subscr_payment has worked flawlessly for us. YMMV, of course, and testing is advised, but based on what we see in these inbound IPNs, the mapping is extremely similar.

Unfortunately, doing this with current versions of WooCommerce Subscriptions requires hacks to the main code. We've opened up an issue asking for help with that.