search

googlesamples / unity-jar-resolver

Unity plugin which resolves Android & iOS dependencies and performs version management
Other
1.21k stars 337 forks source link

Android Resolver version 1.2.173, auto-resolve taking 60 seconds for each jar/aar, while the whole Unity editor unresponsive #551

Closed a-l-e-x-d-s-9 closed 1 year ago

a-l-e-x-d-s-9 commented 1 year ago

[REQUIRED] Please fill in the following fields:

[REQUIRED] Please describe the issue here:

I've upgraded from Unity 2020.3 to Unity 2021.3 and upgraded to the latest version of Android Resolver. Android Resolver starts scanning automatically, and each jar/aar takes 60 seconds, Unity editor is unresponsive. I can't turn off auto-resolve because Unity is unresponsive. Now scanning all files would take a few hours, but it used to be less than five minutes. See the log below. Please help.

Please answer the following, if applicable:

What's the issue repro rate? 100%

Typical looks like that: log Start importing Assets/Plugins/Android/androidx.activity.activity-1.2.0.aar using Guid(57e1baded33ddecaf85bc127b7fc09a0) Importer(-1,00000000000000000000000000000000) -> (artifact id: '064dc319a9dbb6b87344aa39a060dcf5') in 0.044797 seconds Configuring Facebook SDK dlls for each platform Refreshing native plugins compatible for Editor in 11.23 ms, found 3 plugins. Preloading 0 native plugins for Editor in 0.00 ms. Asset Pipeline Refresh: Total: 61.920 seconds - Initiated by StopAssetImportingV2(NoUpdateAssetOptions) Summary: Imports: total=1 (actual=1, local cache=0, cache server=0) Asset DB Process Time: managed=2 ms, native=61521 ms Asset DB Callback time: managed=299 ms, native=96 ms Scripting: domain reloads=0, domain reload time=0 ms, compile time=0 ms, other=0 ms Project Asset Count: scripts=8317, non-scripts=94128 Asset File Changes: new=1, changed=1, moved=0, deleted=0 Scan Filter Count: 4 InvokeBeforeRefreshCallbacks: 0.004ms ApplyChangesToAssetFolders: 0.151ms Scan: 5.550ms OnSourceAssetsModified: 0.069ms InitializeImportedAssetsSnapshot: 111.357ms GetAllGuidsForCategorization: 5.022ms CategorizeAssets: 42.802ms ImportOutOfDateAssets: 61165.924ms (154.886ms without children) ImportManagerImport: 60959.793ms (60914.764ms without children) ImportInProcess: 44.992ms UpdateCategorizedAssets: 0.038ms ReloadImportedAssets: 0.000ms EnsureUptoDateAssetsAreRegisteredWithGuidPM: 25.560ms InitializingProgressBar: 14.673ms PostProcessAllAssetNotificationsAddChangedAssets: 5.460ms OnDemandSchedulerStart: 5.550ms ReloadSourceAssets: 146.416ms UnloadImportedAssets: 1.884ms PostProcessAllAssets: 157.989ms Hotreload: 238.100ms GatherAllCurrentPrimaryArtifactRevisions: 5.132ms UnloadStreamsBegin: 1.535ms LoadedImportedAssetsSnapshotReleaseGCHandles: 1.946ms GetLoadedSourceAssetsIDs: 60816.593ms GetLoadedSourceAssetsSnapshot: 143.197ms PersistCurrentRevisions: 4.447ms UnloadStreamsEnd: 0.410ms Untracked: -60770.734ms Registering precompiled user dll's ... Registered in 0.019475 seconds. Start importing Assets/Plugins/Android/androidx.activity.activity-1.2.0.aar using Guid(57e1baded33ddecaf85bc127b7fc09a0) Importer(-1,00000000000000000000000000000000) -> (artifact id: '16d506409b04dd4d08f14b356f951b96') in 0.058062 seconds Configuring Facebook SDK dlls for each platform Refreshing native plugins compatible for Editor in 11.12 ms, found 3 plugins. Preloading 0 native plugins for Editor in 0.00 ms. Asset Pipeline Refresh: Total: 62.802 seconds - Initiated by StopAssetImportingV2(NoUpdateAssetOptions) Summary: Imports: total=1 (actual=1, local cache=0, cache server=0) Asset DB Process Time: managed=0 ms, native=62457 ms Asset DB Callback time: managed=296 ms, native=46 ms Scripting: domain reloads=0, domain reload time=0 ms, compile time=0 ms, other=0 ms Project Asset Count: scripts=8317, non-scripts=94128 Asset File Changes: new=1, changed=1, moved=0, deleted=0 Scan Filter Count: 4 InvokeBeforeRefreshCallbacks: 0.000ms ApplyChangesToAssetFolders: 0.080ms Scan: 1.535ms OnSourceAssetsModified: 0.066ms InitializeImportedAssetsSnapshot: 106.986ms GetAllGuidsForCategorization: 4.583ms CategorizeAssets: 43.654ms ImportOutOfDateAssets: 62147.761ms (157.190ms without children) ImportManagerImport: 61954.638ms (61896.207ms without children) ImportInProcess: 58.391ms UpdateCategorizedAssets: 0.041ms ReloadImportedAssets: 0.379ms EnsureUptoDateAssetsAreRegisteredWithGuidPM: 24.670ms InitializingProgressBar: 0.003ms PostProcessAllAssetNotificationsAddChangedAssets: 5.565ms OnDemandSchedulerStart: 5.316ms ReloadSourceAssets: 148.085ms UnloadImportedAssets: 1.895ms PostProcessAllAssets: 109.150ms Hotreload: 234.583ms GatherAllCurrentPrimaryArtifactRevisions: 4.022ms UnloadStreamsBegin: 1.968ms LoadedImportedAssetsSnapshotReleaseGCHandles: 2.084ms GetLoadedSourceAssetsIDs: 61777.879ms GetLoadedSourceAssetsSnapshot: 133.409ms PersistCurrentRevisions: 4.331ms UnloadStreamsEnd: 0.477ms Untracked: -61811.850ms Registering precompiled user dll's ... Registered in 0.018480 seconds. Start importing Assets/Plugins/Android/androidx.annotation.annotation-1.1.0.jar using Guid(472e009245c28ec77ab9f63685a57f34) Importer(-1,00000000000000000000000000000000)

google-oss-bot commented 1 year ago

This issue does not seem to follow the issue template. Make sure you provide all the required information.

a-l-e-x-d-s-9 commented 1 year ago

I have also tried older versions: 1.2.172, and 1.2.169, they both act in the same way, with "Asset Pipeline Refresh" taking ~61 seconds. Also, I have tried to use Unity 2021.3.8f1, and it has the same problem. Another issue is that the process always starts from the first file, it never remembers, so if it is interpreted everything will be scanned again. In the log, the problem seems to be with "Asset Pipeline Refresh", "ImportOutOfDateAssets", "ImportManagerImport", that responsible for ~61 seconds delay. I have to use Android Resolver often, because I have to change the package name often, and without Android Resolver, Firebase and Facebook SDK, and other packages, not allowing me to install the app on the device at all. Sometimes I have to change the package name and build packages dozens of times a day, with 2+ hours of Android Resolver addition per change, which makes it impossible. @paulinon Any suggestions on how to overcome this problem, maybe suggestions for workarounds?

@google-oss-bot I've followed the template as best as I could, I don't know what may be the problem that you see.

paulinon commented 1 year ago

Hi @a-l-e-x-d-s-9,

Are you able to untick Enable Auto-Resolution by going to External Dependency Manager > Android Resolver > Settings? Could you provide the complete steps in replicating this behavior along with an output of your Unity Profiler and error logs you may have received? A minimal, reproducible example of your implementation would also be appreciated as this can be used as a baseline for troubleshooting.

a-l-e-x-d-s-9 commented 1 year ago

Hi @paulinon, At first. I wasn't able to turn off Auto-Resolution because Unity Editor is not responsive. But then, I changed the build target from Android to PC, and Android Resolver would not run. Then, I could access the settings and turn off the auto-resolution. When I have changed back to Android build, running Android Resolver > Force Resolve, it is taking ~61 seconds per file, as before. I will try to create minimal, reproducible example, with Unity Profiler and error logs.

a-l-e-x-d-s-9 commented 1 year ago

I have been trying to create a minimal-reproducible project, by installing most packages and plugins that I use. I created the project in Unity 2020.3 and upgraded to 2021.3. In my original project, there are around 105 thousand assets. And the log shows that Asset Pipeline Refresh takes a huge amount of time, so I guess it is related to the number of assets. I have created 100K scriptable objects to add to a minimal-reproducible project. I launched the editor, and Unity was able to create only 44K files, but it would not create more, it continued to run in some sort of loop without producing new files. So I had to remove the files to run a minimal-reproducible project, but could not reproduce the same slow loud behavior as I see in my project. I returned to my original project to record the problem with a profiler, but running Force Resolve caused the profiler to stop recording new frames, so I wasn't able to record anything. I have UnityEditor.Log, with "Asset Pipeline Refresh" taking 54 seconds per each aar/jar file. I have 107 artifacts, so loading all files would take 96 minutes, instead max 5 minutes that it used to take. @paulinon Do think that this very slow load of assets was caused by a bug in Unity 2021.3 and related to 105k assets in my projects? I can open a bug report regarding this issue in the Unity bug report system, not sure what else I can do. Any suggestions?

paulinon commented 1 year ago

Thanks for the update, @a-l-e-x-d-s-9. Let me bring this up to the team so that we can discuss what could be done in this scenario. For now, could you try if version 1.2.174 makes any difference?

a-l-e-x-d-s-9 commented 1 year ago

@paulinon I upgraded to version 1.2.174. It hasn't resolved my problem, I still see a 50-60 seconds delay for a load of each file.

chkuang-g commented 1 year ago

Hi @a-l-e-x-d-s-9

Could you provide us all the ***Dependencies.xml in your project? Also, the content when you click on Asset > External Dependency Manager > Android Resolver > Display Libraries would be helpful for us to debug as well.

At the meantime, I would recommend you to turn on Custom Main Gradle Template and Custome Gradle Properties Template in Player Setting. In this case, EDM4U would just modify mainTemplate.gradle and gradleTemplate.properties which should be much much faster.

Please let us know

a-l-e-x-d-s-9 commented 1 year ago

Here is the AndroidResolverDependencies_xml.log

Here is the Android Resolver - Display Libraries.log

I have turned on Custom Main Gradle Template and Custom Gradle Properties Template in Player Setting, as you suggested. Now when running Android Resolve, it starts resolving Firebase files first, but the import of each file is still 60+ seconds.

@chkuang-g

  1. Can I run into some compatibility problems when upgrading the Unity version while Custom Main Gradle Template and Custom Gradle Properties Template are turned on?
  2. I have already tried to delete my Library and Temp folder, is there another way to clean indexed assets, to make sure it's not some bug in indexing system?
chkuang-g commented 1 year ago

By the import of each file is still 60+ seconds, do you mean importing Android libraries like .aar and .jar?

If so, it is odd because when Custom Main Gradle Template and Custom Gradle Properties Template is enabled, what EDM4U does is just modify mainTemplate.gradle and gradleTemplate.properties, which are just two text files. No import is involved.

And what I meant by ***Dependencies.xml are something like

Just make it faster for us to create an empty project with all these files.

To answer your questions

  1. If by compatibility, you mean dependency resolution, then yes, it is still possible, if the third party plugins in your project depends on vastly different version of the same Android libraries. But at least it should be faster to use the Gradle from Unity to handle this.
  2. By indexing system, do you mean this Index Manager? I am not very familiar with this system. Sometime some package may use ScriptableObject to store cached information under Assets folder. If not, then it is very likely to be under Library/ and Temp/.
a-l-e-x-d-s-9 commented 1 year ago

Here are all *Dependencies.xml that I found: Dependencies.xml.zip. This is what my Player > Build looks like right now: image Here is the latest log of forced Android resolve, ForceAndroidResolve.log, each file taking 60 seconds to resolve. @chkuang-g Please let me know if there is something else I can do.

chkuang-g commented 1 year ago

It is very odd that it can still take 60s after you turn on Custom Main Gradle Template. What EDM4U does is

  1. Copy some .srcaar, which should be in your project already, to be under Assets/GeneratedLocalRepo/
  2. Modify mainTemplate.gradle and gradleTemplate.properties

What took the longest seems to be when EDM4U try to create new folder and when it is doing the copying, like this one Assets/Firebase/m2repository/com/google/firebase/firebase-analytics-unity/9.2.0/firebase-analytics-unity-9.2.0.srcaar. I think the most suspicious part probably this line here. https://github.com/googlesamples/unity-jar-resolver/blob/5c6f23bbc49916b989a20f666a0e0b3306ac9c5d/source/AndroidResolver/src/GradleTemplateResolver.cs#L178

I would be very surprise about Unity if importing a single file can take 60s.

From your case, ImportManagerImport is the one taking the longest and it seems to be a widely-spreaded issue. https://forum.unity.com/threads/incredibly-slow-refresh-editor-completely-freezes.959089/ https://forum.unity.com/threads/need-help-with-incredibly-long-playtime-loading-times.1324332/

Does any of the situation there applies to you?

google-oss-bot commented 1 year ago

Hey @a-l-e-x-d-s-9. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

a-l-e-x-d-s-9 commented 1 year ago

@chkuang-g I was hoping that Unity 2021.3.12f1 would be released with possible changes to my problem, but they didn't release it yet. Little detail about my project, I'm using the same project with different Product Name and Package Name. They share the same base code, and I have a script to change all relevant parameters to switch between different flavors. Is possible that due to changing Product Name and Package Name, Android Resolve has to run extra work? Regarding references to a similar problem from other people, you brought up. The first one is irrelevant to my case. The second one is somewhat similar, but there is no clue of resolution there.

chkuang-g commented 1 year ago

I assume this Product Name and Package Name are probably Unity editor settings, scriptable object or perhaps some C# script?

When you are using mainTemplate.gradle, I do not think changing any of them is gonna trigger Android Resolver to do any heavy lifting stuffs: what it does is basically modifying two files: mainTemplate.gradle and gradleTemplate.properties.

Please let us know if the patch from Unity helps with your situation.

google-oss-bot commented 1 year ago

Hey @a-l-e-x-d-s-9. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

google-oss-bot commented 1 year ago

Since there haven't been any recent updates here, I am going to close this issue.

@a-l-e-x-d-s-9 if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.