Masquerade (out of the box) doesn't anonymise Paypal payment detail

[erfanimani]

I wasn't sure whether this is in scope or not as I'm sure most merchants run third-party payment methods. However, considering that Paypal is shipped with Magento, I probably would have expected this to happen.

[peterjaap]

What tables/columns are those?

[peterjaap]

I wrote the config yml based on what I saw in our clients' databases, and none of our clients use the Paypal option that comes with Magento (it's all through PSPs).

[erfanimani]

Gotcha. I'll investigate which column it is next week. Possibly I can provide a PR myself. From memory however, I think payment information details aren't structured and are stored instead as a json or serialised value.

[peterjaap]

We could add a custom formatter for that; https://github.com/elgentos/masquerade#custom-providers--formatters

[johnorourke]

@erfanimani what's the use case for anonymising this over deleting it? (asking because I'm adding 'delete' options so that we can remove old sales data and reduce DB size for our dev environments, so it would be useful to know if there's a reason for keeping it)

[erfanimani]

@johnorourke no reason! Both options would work.

[peterjaap]

@erfanimani noe that we have the 'delete' option, could you do a PR to add it to the default Magento 2 config?

[erfanimani]

That's great, yeah I should be able to create a PR

[peterjaap]

@erfanimani I just checked a Magento 2 open source database for Paypal and I found no personal identifiable information columns? We don't use Paypal through Magento's implementation (only through payment service providers) so I don't have real data in any of our projects in these tables so I can't be 100% sure.

mysql> show tables like '%paypal%';                                                                                                                                                                                                                                                 [15/251]
+---------------------------------------+                                                                                                     
| Tables_in_magento2m2 (%paypal%) |                                                                                                     
+---------------------------------------+                                                                                                     
| paypal_billing_agreement              |                                                                                                     
| paypal_billing_agreement_order        |                                                                                                     
| paypal_cert                           |                                                                                                     
| paypal_payment_transaction            |                                                                                                     
| paypal_settlement_report              |                                                                                                     
| paypal_settlement_report_row          |                                                                                                     
+---------------------------------------+                                                                                                     
6 rows in set (0.00 sec)                                                                                                                      

mysql> describe paypal_billing_agreement;                                                                                                     
+-----------------+----------------------+------+-----+-------------------+----------------+
| Field           | Type                 | Null | Key | Default           | Extra          |
+-----------------+----------------------+------+-----+-------------------+----------------+
| agreement_id    | int(10) unsigned     | NO   | PRI | NULL              | auto_increment |
| customer_id     | int(10) unsigned     | NO   | MUL | NULL              |                |
| method_code     | varchar(32)          | NO   |     | NULL              |                |
| reference_id    | varchar(32)          | NO   |     | NULL              |                |
| status          | varchar(20)          | NO   |     | NULL              |                |
| created_at      | timestamp            | NO   |     | CURRENT_TIMESTAMP |                |
| updated_at      | timestamp            | YES  |     | NULL              |                |
| store_id        | smallint(5) unsigned | YES  | MUL | NULL              |                |
| agreement_label | varchar(255)         | YES  |     | NULL              |                |
| agreement_data  | text                 | YES  |     | NULL              |                |
+-----------------+----------------------+------+-----+-------------------+----------------+
10 rows in set (0.00 sec)

mysql> describe paypal_billing_agreement_order;
+--------------+------------------+------+-----+---------+-------+
| Field        | Type             | Null | Key | Default | Extra |
+--------------+------------------+------+-----+---------+-------+
| agreement_id | int(10) unsigned | NO   | PRI | NULL    |       |
| order_id     | int(10) unsigned | NO   | PRI | NULL    |       |
+--------------+------------------+------+-----+---------+-------+
2 rows in set (0.00 sec)

mysql> describe paypal_payment_transaction;
+------------------------+------------------+------+-----+---------+----------------+
| Field                  | Type             | Null | Key | Default | Extra          |
+------------------------+------------------+------+-----+---------+----------------+
| transaction_id         | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| txn_id                 | varchar(100)     | YES  | UNI | NULL    |                |
| additional_information | blob             | YES  |     | NULL    |                |
| created_at             | timestamp        | YES  |     | NULL    |                |
+------------------------+------------------+------+-----+---------+----------------+
4 rows in set (0.00 sec)

mysql> describe paypal_settlement_report;
+---------------+------------------+------+-----+---------+----------------+
| Field         | Type             | Null | Key | Default | Extra          |
+---------------+------------------+------+-----+---------+----------------+
| report_id     | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| report_date   | date             | YES  | MUL | NULL    |                |
| account_id    | varchar(64)      | YES  |     | NULL    |                |
| filename      | varchar(24)      | YES  |     | NULL    |                |
| last_modified | timestamp        | YES  |     | NULL    |                |
+---------------+------------------+------+-----+---------+----------------+
5 rows in set (0.00 sec)

mysql> describe paypal_settlement_report_row;
+-----------------------------+------------------+------+-----+----------+----------------+
| Field                       | Type             | Null | Key | Default  | Extra          |
+-----------------------------+------------------+------+-----+----------+----------------+
| row_id                      | int(10) unsigned | NO   | PRI | NULL     | auto_increment |
| report_id                   | int(10) unsigned | NO   | MUL | NULL     |                |
| transaction_id              | varchar(19)      | YES  |     | NULL     |                |
| invoice_id                  | varchar(127)     | YES  |     | NULL     |                |
| paypal_reference_id         | varchar(19)      | YES  |     | NULL     |                |
| paypal_reference_id_type    | varchar(3)       | YES  |     | NULL     |                |
| transaction_event_code      | varchar(5)       | YES  |     | NULL     |                |
| transaction_initiation_date | timestamp        | YES  |     | NULL     |                |
| transaction_completion_date | timestamp        | YES  |     | NULL     |                |
| transaction_debit_or_credit | varchar(2)       | NO   |     | CR       |                |
| gross_transaction_amount    | decimal(20,6)    | NO   |     | 0.000000 |                |
| gross_transaction_currency  | varchar(3)       | YES  |     | NULL     |                |
| fee_debit_or_credit         | varchar(2)       | YES  |     | NULL     |                |
| fee_amount                  | decimal(20,6)    | NO   |     | 0.000000 |                |
| fee_currency                | varchar(3)       | YES  |     | NULL     |                |
| custom_field                | varchar(255)     | YES  |     | NULL     |                |
| consumer_id                 | varchar(127)     | YES  |     | NULL     |                |
| payment_tracking_id         | varchar(255)     | YES  |     | NULL     |                |
| store_id                    | varchar(50)      | YES  |     | NULL     |                |
+-----------------------------+------------------+------+-----+----------+----------------+

[erfanimani]

Thanks for checking @peterjaap.

So I'm talking about this part here:

It seems it's JSON serialised data stored in sales_order_payment/additional_information

It seems we can't truncate that table as it'll cause an error Call to a member function getAdditionalInformation() on null.

I can create a PR that adds a config to simply add an empty object in the additional_data column. It also has some cc fields it seems, but I don't think they're used. How does that sound? CC @johnorourke

[peterjaap]

Sounds good, you can do that with the formatter: fixed formatter, see an example here: https://github.com/elgentos/masquerade/blob/6bd9f9275b3ce16fa71d22ed5b3d55c43763b2a3/src/config/magento2/customer.yaml#L10

[johnorourke]

@erfanimani thanks :)