search

oracle / cordova-plugin-wkwebview-file-xhr

Cordova Plugin for WebView File XHR
Universal Permissive License v1.0
138 stars 120 forks source link

How to use with third-party software? #16

Closed vortice3D closed 6 years ago

vortice3D commented 6 years ago

Hi there:

The case is that I'm using a webgl engine (BabylonJS) in one of my cross-platform developments.

My code loads textures, in the form of PNG files, using a specific API, like this:

texGeneric=new BABYLON.Texture("assets/textures/Generic.png",_Scene);
...

This works perfectly.

Also, I need to load a cube map/environment in the form of a DDS file, using this other API:

texEnvironment=BABYLON.CubeTexture.CreateFromPrefilteredData("assets/textures/environment.dds",_Scene);

This doesn't work.

Internally CreateFromPrefilteredData uses XHR, so when ported to a plain Cordova-iOS it can´t find the DDS file, as WKWebView appears to treat local files as if they came from a remote server, even though they're in the app itself, and such requests are blocked.

In this case I thought cordova-plugin-wkwebview-file-xhr could make it work but, when used, the app can't find the (previously working) local files as it output in the remote debug console from Safari: Cross origin requests are only supported for HTTP.

I have also this relevant output in Xcode: 2018-04-02 12:15:30.245199+0200 GRB Technology[1165:700339] Could not signal service com.apple.WebKit.WebContent: 113: Could not find specified service 2018-04-02 12:15:30.248572+0200 GRB Technology[1165:700339] Could not signal service com.apple.WebKit.Networking: 113: Could not find specified service

Any clue on this? Do I need to add "http://" or something at the beginning of PNG paths?

Thanks for your time.

gvanmat commented 6 years ago

@vortice3D The cordova-plugin-wkwebview-file-xhr has a restriction of only allowing readonly access to files under the "www" folder of the applications bundle. This is a signed secure part of the file system that cannot be altered once the application is installed. Is the "asserts" folder located under "www" within the cordova application or is the file located on a remote endpoint?

Try enabling "full" plugin logging. The logging is made to the JavaScript console log of the webview.

vortice3D commented 6 years ago

Hi gvanmat, and thank your for your time.

My folder tree is this: www └-----/assets └-----/libs └-----/scripts └-----/styles └ index.html

Best regards.

gvanmat commented 6 years ago

@vortice3D The file structure looks good. You will need to enable full logging to give some insight into what's failing.

vortice3D commented 6 years ago

Well, gvanmat...excuse my lack of experience in this but, how can I enable such a "full" plugin logging?

Must I use "cordova build iOS --verbose"?

vortice3D commented 6 years ago

Excuse me. Wrong button!!

gvanmat commented 6 years ago

@vortice3D You need to edit your config.xml found in the same root folder as www adding the following parameter:

<preference name="NativeXHRLogging" value="full"/>

vortice3D commented 6 years ago

Thanks gvanamt.

Is estrange, but the addition of the preference directive seems to do nothing, as I get the same remote console log than I got before (but not show in this thread):

[Error] XMLHttpRequest cannot load file:///var/containers/Bundle/Application/3685B4EA-BF39-4ABD-90B1-514825761447/GRB%20Technology.app/www/assets/models/Monomando.babylon.manifest?1522682768175. Cross origin requests are only supported for HTTP.
    checkManifestFile (babylon.3.1.custom.min.js:32:24533)
    t (babylon.3.1.custom.min.js:32:23392)
    _loadData (babylon.3.1.custom.min.js:32:2007)
    Append (babylon.3.1.custom.min.js:32:3894)
    _create (vortice3d.grb-bjs.js:121)
    Engine3D (vortice3d.grb-bjs.js:845)
    (función anónima) (grb.js:57)
[Error] XMLHttpRequest cannot load file:///var/containers/Bundle/Application/3685B4EA-BF39-4ABD-90B1-514825761447/GRB%20Technology.app/www/assets/models/Termostatico.babylon.manifest?1522682768176. Cross origin requests are only supported for HTTP.
    checkManifestFile (babylon.3.1.custom.min.js:32:24533)
    t (babylon.3.1.custom.min.js:32:23392)
    _loadData (babylon.3.1.custom.min.js:32:2007)
    Append (babylon.3.1.custom.min.js:32:3894)
    _create (vortice3d.grb-bjs.js:122)
    Engine3D (vortice3d.grb-bjs.js:845)
    (función anónima) (grb.js:57)
[Error] XMLHttpRequest cannot load file:///var/containers/Bundle/Application/3685B4EA-BF39-4ABD-90B1-514825761447/GRB%20Technology.app/www/assets/models/Monomando.babylon.manifest. Cross origin requests are only supported for HTTP.
    (función anónima) (babylon.3.1.custom.min.js:32:24505)
[Error] XMLHttpRequest cannot load file:///var/containers/Bundle/Application/3685B4EA-BF39-4ABD-90B1-514825761447/GRB%20Technology.app/www/assets/models/Termostatico.babylon.manifest. Cross origin requests are only supported for HTTP.
    (función anónima) (babylon.3.1.custom.min.js:32:24505)
[Error] XMLHttpRequest cannot load file:///var/containers/Bundle/Application/3685B4EA-BF39-4ABD-90B1-514825761447/GRB%20Technology.app/www/assets/models/Monomando.babylon. Cross origin requests are only supported for HTTP.
    h (babylon.3.1.custom.min.js:3:12095)
    LoadFile (babylon.3.1.custom.min.js:3:12366)
    _ (babylon.3.1.custom.min.js:32:1848)
    i (babylon.3.1.custom.min.js:32:23856)
    (función anónima) (babylon.3.1.custom.min.js:32:24514)
[Error] XMLHttpRequest cannot load file:///var/containers/Bundle/Application/3685B4EA-BF39-4ABD-90B1-514825761447/GRB%20Technology.app/www/assets/models/Termostatico.babylon. Cross origin requests are only supported for HTTP.
    h (babylon.3.1.custom.min.js:3:12095)
    LoadFile (babylon.3.1.custom.min.js:3:12366)
    _ (babylon.3.1.custom.min.js:32:1848)
    i (babylon.3.1.custom.min.js:32:23856)
    (función anónima) (babylon.3.1.custom.min.js:32:24514)
[Error] BJS - [17:26:08]: Unable to load from ./assets/models/Monomando.babylon: 0 
    (función anónima) (cordova.js:1731)
    _ErrorEnabled (babylon.3.1.custom.min.js:3:19303)
    c (babylon.3.1.custom.min.js:32:3794)
    (función anónima) (babylon.3.1.custom.min.js:32:1878)
    onreadystatechange (babylon.3.1.custom.min.js:3:12081)
[Error] BJS - [17:26:08]: Unable to load from ./assets/models/Termostatico.babylon: 0 
    (función anónima) (cordova.js:1731)
    _ErrorEnabled (babylon.3.1.custom.min.js:3:19303)
    c (babylon.3.1.custom.min.js:32:3794)
    (función anónima) (babylon.3.1.custom.min.js:32:1878)
    onreadystatechange (babylon.3.1.custom.min.js:3:12081)
[Error] ReferenceError: Can't find variable: nulll
    _createScene (vortice3d.grb-bjs.js:209)
    notifyObservers (babylon.3.1.custom.min.js:3:3626)
    _checkIsReady (babylon.3.1.custom.min.js:11:23164)
    (función anónima) (babylon.3.1.custom.min.js:11:23041)

Only as a help on understandingd that, the case is that this WebGL Engine uses a specific JSON file in order to describe scenes (meshes + lights + animations +...). In my code I have two of these JSONs, namely Monomando.babylon and Temostatico.babylon.

Well now (with the use of cordova-plugin-wkwebview-file-xhr) I can´t load them, but remember that before adding this plugin I could load them, and also the PNG textures, but not the CubeMap style DDS textures (as this last is loaded by the WebGL Engine using a XHR mechanism). This last issue was the problem who bring me to use cordova-plugin-wkwebview-file-xhr.

gvanmat commented 6 years ago

@vortice3D Sounds like you are going to need to do some debugging to determine why the cordova-plugin-wkwebview-file-xhr plugin is not being loaded. It might also be that the babylon script library is loaded before the plugin-wkwebview-file-xhr plugin. You should be able to attach the Safari browser to the application running in the XCode IDE.

vortice3D commented 6 years ago

I see.

In index.html I've just changed the to a position before the , and I get this kind of console output (excerpt):

"XMLHttpRequest cannot load file:///var/containers/Bundle/Application/E00CC1C9-7EB4-4828-826B-8270625C142C/GRB%20Technology.app/www/assets/models/Monomando.babylon.manifest?1522685023139. Cross origin requests are only supported for HTTP."

So clearly we have here a COR issue.

vortice3D commented 6 years ago

BTW, reviewing my config.xml file, I can see the line <plugin name="cordova-plugin-wkwebview-file-xhr" spec="^2.1.1" /> but nothing about the dependent cordova-plugin-wkwebview.

Anyway, the two plugins are listed when a cordova plugin list is done.

gvanmat commented 6 years ago

@vortice3D I think the issue is the babylon APIs are using synchronous uses of XMLHttpRequest versus async. I see places in the code like:

var i = new XMLHttpRequest;
i.open("get", r.href, false);
var s = i.responseText.replace....

The third argument async has a value of "false".

The XMHttpRequest polyfill doesn't currently support synchronous calls. However, it might be possible to implement with Safari 10.1 supporting await.

gvanmat commented 6 years ago

@vortice3D The snippet of code I posted before is loading CSS. I don't think the images are loaded via XMLHttpRequest but rather some other technique using the browser. Whatever method is being used by this library to load images, it's not XMLHttpRequest or the Fetch API.

Unfortunately, fixing the synchronous send is not going to be a complete solution.

gvanmat commented 6 years ago

@vortice3D I debugged thru the babylon.js some and found the method it's using to load images. The method is LoadImage. They are creating an Javascript Image Object, setting the src to the target URI and adding "load" event listeners. This would be outside the scope of the cordova-plugin-wkwebview-file-xhr.

vortice3D commented 6 years ago

Hi gvanmat:

First of all I wish to thank you very much your interest and support with this issue.

Well, I know BJS is using that LoadImage API in order to "load the plain images", when I call BABYLON.Texture but as said I can load that textures (PNG files in my case) w/o problems under browser, Cordova-Android and Cordova-iOS platform.

The problems arises (only on Cordova-iOS), with the loading of environment textures, under the form of a DDS file, by means of BABYLON.CubeTexture.CreateFromPrefilteredData. BJS guys said me this behavior is because the engine is using an XHR loading mechanism in this case (logical as a DDS is dealing with a cubemap of six images) and Cordova-iOS or maybe iOS-UIWebView-WKWebView shows a malfunction in this case.

BTW I don´t know if it could be enough to use some of these plugins:

Well, I'll let you know any advance on this.

Thanks for your time.

gvanmat commented 6 years ago

@vortice3D

I was able to create a sample cordova ios app using the wkwebview-file-xhr plugin with a sample babylonJS app that uses BABYLON.SceneLoader.Load("textures/", "v8.babylon", engine, ...

The XMLHttpRequest polyfill provided by the plugin is intercepting the file:// GET requests. I've attached a zip of the www folder of my project. You might compare the script includes from index.html.

www.zip

vortice3D commented 6 years ago

Hi Gary:

Thank for your collaborarion on all this.

I'll inspect your zip tomorrow and let you know.

Good night.

vortice3D commented 6 years ago

Hi again Gary:

I've seen your example of use.

Well, you're loading a .babylon file (scene) that is only a text file (JSON). I've not found problems loading local JSONs or PNGs, but I can´t load local DDS or MP3, as the WebGL Engine uses XHR in order to load such a binary contents.

This way, if I host those DDS or MP3 in my server, and Ioad then from it in my code, then all works OK. But I'd prefer to have it stored locally, and so no external communications would be necessary for my app.

The fact is that XHR doesn't work on local contention on iOS, as it expects a remote served content for using Ajax calls, refusing local ones. After all, that's because ""cordova plugin add cordova-plugin-wkwebview-file-xhr" plugin has been developed, isn't it?

Anyway, after adding the plugin by means of "cordova plugin add cordova-plugin-wkwebview-file-xhr", the previously loaded OK PNGs are now not found, getting this Xcode errors:

2018-04-04 11:22:11.648166+0200 GRB Technology[1492:1037474] Using WKWebView
2018-04-04 11:22:11.649114+0200 GRB Technology[1492:1037474] [CDVTimer][console] 0.279903ms
2018-04-04 11:22:11.649489+0200 GRB Technology[1492:1037474] [CDVTimer][handleopenurl] 0.232935ms
2018-04-04 11:22:11.652544+0200 GRB Technology[1492:1037474] [CDVTimer][intentandnavigationfilter] 2.942085ms
2018-04-04 11:22:11.652795+0200 GRB Technology[1492:1037474] [CDVTimer][gesturehandler] 0.142097ms
2018-04-04 11:22:11.671770+0200 GRB Technology[1492:1037474] [CDVTimer][splashscreen] 18.877983ms
2018-04-04 11:22:11.683911+0200 GRB Technology[1492:1037474] [CDVTimer][statusbar] 11.906981ms
2018-04-04 11:22:11.685166+0200 GRB Technology[1492:1037474] [CDVTimer][cdvwkwebviewfilexhr] 1.095057ms
2018-04-04 11:22:11.685278+0200 GRB Technology[1492:1037474] [CDVTimer][TotalPluginStartup] 36.586046ms
2018-04-04 11:22:11.752325+0200 GRB Technology[1492:1037474] Could not signal service com.apple.WebKit.WebContent: 113: Could not find specified service
2018-04-04 11:22:11.757525+0200 GRB Technology[1492:1037474] Could not signal service com.apple.WebKit.Networking: 113: Could not find specified service
2018-04-04 11:22:13.756591+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/tube/Tubo_Metallic_PBR.png
2018-04-04 11:22:13.756999+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/tube/Tubo_Normal.png
2018-04-04 11:22:13.757426+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/mono/Manetas_BaseColor.png
2018-04-04 11:22:13.757768+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/mono/Manetas_Metallic_PBR.png
2018-04-04 11:22:13.758363+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/mono/Resto_Normal.png
2018-04-04 11:22:13.759055+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/mono/Manetas_Normal.png
2018-04-04 11:22:13.759380+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/thermo/Manetas_BaseColor.png
2018-04-04 11:22:13.759690+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/thermo/Manetas_Metallic_PBR.png
2018-04-04 11:22:13.759907+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/thermo/Resto_Normal.png
2018-04-04 11:22:13.760240+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/thermo/Manetas_Normal.png
2018-04-04 11:22:13.760517+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/thermo/Gizmos.png
2018-04-04 11:22:13.760778+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/512/hand.png
2018-04-04 11:22:13.761168+0200 GRB Technology[1492:1037474] ERROR: BJS - [11:22:13]: Error while trying to load image: assets/textures/256/rain.png
vortice3D commented 6 years ago

BTW, reviewing the header's metas in your index.html:

<meta http-equiv="Content-Security-Policy" content="default-src * data: *; style-src *; media-src *; img-src * data: * content: *;">
<meta name="format-detection" content="telephone=no">
<meta name="msapplication-tap-highlight" content="no">

And visiting then the https://github.com/apache/cordova-plugin-whitelist/blob/master/README.md#content-security-policy I can read that: "Controls which network requests (images, XHRs, etc) are allowed to be made (via webview directly)".

So is this the solution to all this? I mean, not using at all the "cordova plugin add cordova-plugin-wkwebview-file-xhr" plugin but only tweaking the whitelist policies?

gvanmat commented 6 years ago

@vortice3D The "Content-Security-Policy" controls what resources the browser can load such as img, style, script, and media. XHR is invoked via JavaScript thru either the XMLHttpRequest or the Fetch API. The plugin is a workaround for the CORS constraints for XHR (XMLHttpRequest and Fetch API) only.

If you debug the Babylon APIs, loading of the images are not going thru XHR. They creating a Image object and assigning the src to the local resource. I suspect the media is doing they same type of thing. You might ask the Babylon guys if there is a way to form the URL so the local resource can be loaded via XHR. I think if you use file:// it will load the resource using the File API and assign the image as a data url of the image, which would mean you would need the cordova-plugin-file.

One more point on the wkwebivew-file-xhr plugin, by default only secure "https" remote calls are intercepted. You have to set the InterceptRemoteRequests preference to "all".

Please also note that in the sample I provided where loading the v8.babylon file is being intercepted by the wkwebview-file-xhr plugin, the babylon script is not invoked until the device is ready. See the /js/index.js. If you are not already doing this, you will want to wait for cordova to be ready before running additional scripts.

vortice3D commented 6 years ago

Hi Gary:

I agree with you in the way BJS is loading contents, standard image files are managed through an Image, but other (binary) types needs to use Ajax calls (XHR) as no one HTML-DOM element can load it.

Asking the BJS guys about this issue they are saying me it's a bug of Cordova with local XHR (and that's the reason I got initially to this Cordova plugin).

I've tried the "cordova-plugin-file" + "file://" combination w/o success:

018-04-04 16:30:06.144685+0200 GRB Technology[1540:1057567] NSURLConnection finished with error - code -1100
2018-04-04 16:30:06.151657+0200 GRB Technology[1540:1057326] ERROR: BJS - [16:30:06]: error on XHR request.

Well, I really appreciate your support on this, but as I do not wish to take more time from you, and this is "not-a-really-this-plugin-issue" (but Cordova's one) I think the best is to use remote served media (as Cordova-iOS expect from XHR) and close the issue.

Thanks.

vortice3D commented 6 years ago

Well, I've done a last intent, following your advice of create the BJS scene after device-ready is raised (I was doing it on DOMContentLoaded as, well, is the way of doing it for browsers) and now (finally) the DDS is loaded.

Sadly, is not the same story with the MP3, as the Xcode console outputs:

2018-04-04 16:43:53.758536+0200 GRB Technology[1558:1061684] ERROR: BJS - [16:43:53]: error on XHR request.
2018-04-04 16:43:53.760983+0200 GRB Technology[1558:1061684] xhr-polyfill.js - exception delivering event {"isTrusted":false} TypeError: Argument 1 ('audioData') to webkitAudioContext.decodeAudioData must be an instance of ArrayBuffer

So, how can I add a Trust Policy for MP3s in my app?

gvanmat commented 6 years ago

The MP3 load issue is likely because the babylon code is not setting the XMLHttpRequest responseType to "arraybuffer". The polyfill with this plugin only has a small list of file suffixes to default response types. The native XMLHttpRequest probably defaults "arraybuffer" based on the URL suffix but we are defaulting to text.

vortice3D commented 6 years ago

Hi Gary:

Only if this info can be useful to you or somebody else, I've asked BJS staff about your thoughts on the responseType on XHR call for:

BABYLON.Sound("sfx","assets/sounds/sfx.mp3",_Scene,null,{
  autoplay: false,
  loop: true,
  volume: 0}
);

and they've answered that's weirds, as "...because we ARE actually setting the responseType to arrayBuffer: https://github.com/BabylonJS/Babylon.js/blob/c224954e1b2005f1d01fa711904075e0129a208e/src/Tools/babylon.tools.ts#L604"