CSP errors still here and no records in reports/rules on admin side

superdevofficial commented 4 years ago

Hello,

I have set it up with composer, done some cli commands as setup:upgrade, di:compile, cache:clean... The module is enabled in Stores / Configuration / Security / CSP, with report only. Cron is enabled and I have tried to run it.

I still have these CSP errors (example) : [Report Only] Refused to load the stylesheet 'https://fonts.googleapis.com/css?family=Work+Sans:400,700.less' because it violates the following Content Security Policy directive: "style-src getfirebug.com 'self' 'unsafe-inline'". Note that 'style-src-elem' was not explicitly set, so 'style-src' is used as a fallback.

No records in the reports or the rules on admin side...but i can see datas in the database. With a select * from fl32_csp_reports, I find 24 rows and this is the last one : | 24 | 2020-05-27 09:01:42 | 1 | {"document-uri":"http://terre-exotique.local/adminsd/fl32csp/rule/listing/key/7d6bea936c6426569c40e66378fcba7d2659bbad89dbde31d78cd94153b1e175/","referrer":"http://terre-exotique.local/adminsd/fl32csp/report/listing/key/cf19f5cfda058a35e709abbdfda3c76ecc27e50850beead191d72f2ac6f5d2f6/","violated-directive":"font-src","effective-directive":"font-src","disposition":"report","blocked-uri":"https://fonts.gstatic.com/s/worksans/v7/QGYsz_wNahGAdqQ43Rh_cqDpp_k.woff2","status-code":200,"script-sample":""}

If you have one idea about that ? ^^'

I work on a local dev setup, located in a docker container with MariaDb:10.2 and Magento version:2.3.5-p1

Best regards, A Junior Magento2 Dev

flancer64 commented 4 years ago

Hello @superdevofficial !

This is second case when DB data is not available in admin grids, but I cannot reproduce the situation locally. If you have data in fl32_csp_report you can run analyzer manually:

$ ./bin/magento fl32:csp:analyze

I suppose you should have new records in fl32_csp_rule after that.

Could you run CLI command and send any record from fl32_csp_rule to me?

Thanks, Alex.

superdevofficial commented 4 years ago

Thank you for the CLI tips.

First thing, when done the bin/magento fl32:csp:analyze

Command 'fl32:csp:analyze': Analyze CSP violation reports and compose policy rules.
2 rules were added. 24 reports were deleted (id: 1-24).
Command 'fl32:csp:analyze' is completed.

And the 2 rows added in rules :

Select * from fl32_csp_rule;
+----+---------+------------+---------+----------------------+
| id | enabled | admin_area | type_id | source               |
+----+---------+------------+---------+----------------------+
|  1 |       1 |          1 |      22 | fonts.googleapis.com |
|  2 |       1 |          1 |       5 | fonts.gstatic.com    |
+----+---------+------------+---------+----------------------+

The rules seems to work, because I no more have the JS errors but nothing appears on the admin side :/

Joso commented 4 years ago

Hi Flancer64, I confirm this issue on Magento 2.3.5-p1

flancer64 commented 4 years ago

Thanks, guys.

I'll investigate the problem. I don't know the possible reason for the now :( I need some time.

Joso commented 4 years ago

Thanks, guys.

I'll investigate the problem. I don't know the possible reason for the now :( I need some time.

Hi flancer64 on y site have conflict detector, and report this;

Original Class	Rewrites	Conflict
Flancer32\Csp\Api\Repo\Dao\Rule	Flancer32\Csp\Repo\Dao\Rule	NO
Flancer32\Csp\Api\Repo\Dao\Type\Policy	Flancer32\Csp\Repo\Dao\Type\Policy	NO
Flancer32\Csp\Api\Repo\Dao\Report	Flancer32\Csp\Repo\Dao\Report

I think there is a likely conflict with the classes. If this can help you....

Vlad242 commented 4 years ago

Hi Flancer64, I also tried your module and came across a similar problem. You wrote in the DataProvider code about "hard-coded arguments", in new versions of magento the new style of writing of Ui-components was added, here is one of the examples of listing components:

<argument name="data" xsi:type="array">
        <item name="js_config" xsi:type="array">
            <item name="provider" xsi:type="string">chat_listing.chat_listing_data_source</item>
        </item>
    </argument>
    <settings>
        <spinner>chat_columns</spinner>
        <deps>
            <dep>chat_listing.chat_listing_data_source</dep>
        </deps>
    </settings>
    <dataSource name="chat_listing_data_source" component="Magento_Ui/js/grid/provider">
        <settings>
            <storageConfig>
                <param name="indexField" xsi:type="string">chat_hash</param>
            </storageConfig>
            <updateUrl path="mui/index/render"/>
        </settings>
        <aclResource>Vendor_ChatWidget::listing</aclResource>
        <dataProvider
                class="Magento\Framework\View\Element\UiComponent\DataProvider\DataProvider"
                name="chat_listing_data_source">
            <settings>
                <requestFieldName>chat_id</requestFieldName>
                <primaryFieldName>chat_id</primaryFieldName>
            </settings>
        </dataProvider>
    </dataSource>

maybe this code can help you)

flancer64 commented 4 years ago

Hello, guys.

I cannot reproduce this bug, so I've added logging features to my base data provider for grids - \Flancer32\Base\App\Ui\DataProvider\Admin\Grid

Could somebody replace ./vendor/flancer32/mage2_ext_base/App/Ui/DataProvider/Admin/Grid.php with patched version, open rules grid (with data in DB) in adminhtml and filter log records:

$ cat ./var/log/system.log | grep "fl32csp_log: " > fl32csp.grid.log

then email fl32csp.grid.log file to me (alex@flancer64.com)

Thanks, Alex.

superdevofficial commented 4 years ago

OK - Log sent ;)

By the way, thank you for this module ;)

flancer64 commented 4 years ago

Thank you, @superdevofficial !

I see that my provider selects data from DB and sends it to the caller (\Magento\Framework\View\Element\UiComponent\Context::getDataSourceData).

The possible reason of the failure may be the fact that selected data has identity filed named as Id (query, grid definition) but data provider has primary & request fields named as id (as mentioned by @Vlad242 ).

To check this assumption we need to change

<column name="Id"

to

<column name="id"

in ./vendor/flancer32/mage2_ext_csp/view/adminhtml/ui_component/fl32csp_rule_grid.xml

and change

const A_ID = 'Id';

to

const A_ID = 'id';

in ./vendor/flancer32/mage2_ext_csp/Ui/DataProvider/Rule/Grid.php

flancer64 commented 4 years ago

Another assumption is the edition of Magento. I tested this extension with Magento 2.3.5-p1 Open Source only.

@superdevofficial , @Joso , @Vlad242 , do you have the empty grids with Open Source edition or with Magento Commerce edition?

Joso commented 4 years ago

Hi @flancer64, in my magento admin the the csp_report in empty, too in DB; the csp_rule in magento admin in empty but in DB there are 9 record:

idRule ID.	enabledIs policy rule enabled?	admin_areaIs policy rule applied to admin or storefront area…	type_idPolicy type ID.	sourceSource value (schema, URL, hash, ...).
1	1	0	9	www.facebook.com
2	1	1	9	cm.everesttech.net
3	1	1	8	amc.demdex.net
4	1	1	9	dpm.demdex.net
5	0	1	19	channels.magento.com
6	0	1	3	channels.magento.com
7	0	1	22	channels.magento.com
8	0	1	19	cdn.wootric.com
9	0	1	3	eligibility.wootric.com

Magento 2.3.5-P1 Open Source (or Community edition), php 7.3.18, Contabo VPS 8gb ram OS Debian 9 and Plesk

Joso commented 4 years ago

Hi @flancer64 , I tried the module on another magento 2.3.5-p1 site, after installing the module, both frontend and backend goes in time out. The pages have not been loaded. The site hosted on different VPS than the other site. but the system is the same: Debian 9, Plesk obsidian, 8Gb ram.

Could it be that the module requires a lot of resources, and with continuous use? Because on this server I also have another site in Magento 1 and both sites were unreachable at the same time. They came back live after I disabled the module, so I removed it from the site. I tried it two time.

flancer64 commented 4 years ago

Hi, @Joso .

This module should not requires a lot of resources but I don't tested it on highly loaded apps. Each web page can generate a lot of CSP violation reports and every report is a single request to the server and a single record into DB. Do you have records in fl32_csp_report and how many records do you have there?

I suppose, you need run CLI analyzer just after installation to generate the most popular rules to eliminate number of CSP reports in this case.

$ ./bin/magento fl32:csp:analyze

This will add rules to the CSP header and reduce number of reports immediately.

Joso commented 4 years ago

Hi, I tested it, this is the result: Command 'fl32:csp:analyze': Analyze CSP violation reports and compose policy rules. 3 rules were added. 3 reports were deleted (id: 1-3). Command 'fl32:csp:analyze' is completed. but the sites are going in timeout again, could it be a conflict with other extensions?

flancer64 commented 4 years ago

@Joso , let jump to the other issue (#4) to solve server freeze problem.

Vlad242 commented 4 years ago

Hi, @flancer64, I replaced the provider code, checked the logs and did not receive any entry in the log file, in the system logs I received only:

[2020-06-01 07:17:34] main.INFO: Add of item with id Flancer32_Csp::menu was processed [] []
[2020-06-01 07:17:34] main.INFO: Add of item with id Flancer32_Csp::report_listing was processed [] []
[2020-06-01 07:17:34] main.INFO: Add of item with id Flancer32_Csp::rule_listing was processed [] []

but the list of rules in the corresponding field of the admin panel did not appear (

Regards, @Vlad242

superdevofficial commented 4 years ago

Hello everybody,

I have looked at the XHR requests :

Request : ?namespace=fl32csp_rule_grid&sorting%5Bfield%5D=Id&sorting%5Bdirection%5D=desc&filters%5Bplaceholder%5D=true&paging%5BpageSize%5D=20&paging%5Bcurrent%5D=1&isAjax=true

Response :

{
"error":true,
"message":"You entered an invalid Secret Key. Please refresh the page."
}

If you want another data, just ask... I don't know which secret key is required here. There is my "cookie data" :

For request :

{
"admin": "35aac8ab9c26914bbef4db037ab85643",
"AMCV_8F99160E571FC0427F000101@AdobeOrg": "XXX***XXX", // adobe personal data erased ;)
"form_key": "0qd1698SHufvRb1b",
"mage-cache-sessid": "true",
"mage-cache-storage": "{}",
"mage-cache-storage-section-invalidation": "{}",
"mage-messages": "",
"private_content_version": "50d078f46af7afad8f659c5e450af58f",
"product_data_storage": "{}",
"recently_compared_product": "{}",
"recently_compared_product_previous": "{}",
"recently_viewed_product": "{}",
"recently_viewed_product_previous": "{}"
}

For response :

{
"admin": {
"domain": "my-magento.local",
"expires": "2021-06-02T07:58:31.000Z",
"httpOnly": true,
"path": "/adminsd",
"value": "35aac8ab9c26914bbef4db037ab85643"
},
"form_key": {
"domain": "my-magento.local",
"expires": "2021-06-02T07:58:31.000Z",
"path": "/adminsd",
"value": "0qd1698SHufvRb1b"
}
}

flancer64 commented 4 years ago

Thanks, guys! I've set "Config / Advanced / Admin / Security / Add Secret Key to URLs" to "Yes" and now I can reproduce the situation.

flancer64 commented 4 years ago

Thanks a lot to all!

The reason was I used \Magento\Framework\UrlInterface instead of \Magento\Backend\Model\UrlInterface and getRouteUrl instead of getUrl.

Please run

$ composer update flancer32/mage2_ext_base

to refresh existing deployment. Version 0.2.3 of flancer32/mage2_ext_base has no this error.

flancer32 / mage2_ext_csp

CSP errors still here and no records in reports/rules on admin side #2