Appstraction download hangs and does not complete

synapticlee commented 1 year ago

Hi, downloading appstraction in both npm and yarn is hanging, this is the output from yarn:

yarn add v1.22.19
warning package.json: No license field
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
warning No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning "appstraction > ts-node@10.9.1" has unmet peer dependency "@types/node@*".
warning "appstraction > ts-node@10.9.1" has unmet peer dependency "typescript@>=2.7".
warning "appstraction > ipa-extract-info > zipjs-browserify > browserify-shim@3.8.16" has unmet peer dependency "browserify@>= 2.3".
[4/4] 🔨  Building fresh packages...
[1/4] ⠂ frida
[2/4] ⠂ cpu-features
[-/4] ⠂ waiting...
warning Error running install script for optional dependency: "/Users/jlee/node_modules/cpu-features: Command failed.
Exit code: 127
Command: node buildcheck.js > buildcheck.gypi && node-gyp rebuild
Arguments:
Directory: /Users/jlee/node_modules/cpu-features
[4/4] ⢀ appstraction

baltpeter commented 1 year ago

The first time you download appstraction, it has to download a whole bunch of dependencies. That can take a few minutes. That's probably what's happening.

But I can absolutely see how that looks like it's hanging. We should add some output that informs the user that this can take a while without any output.

synapticlee commented 1 year ago

I waited a considerable number of minutes but I can try it again - I think I was waiting for at least 10-30 minutes at that point!

baltpeter commented 1 year ago

Oh, that's a lot. :o

@zner0L How long did it take on your MacBook?

synapticlee commented 1 year ago

Likewise installing via npm is taking forever (more than 10-30 mins by now) and hanging after the step: (##################) ⠹ reify:typescript: timing reifyNode:node_modules/typescript Completed in 3013ms

zner0L commented 1 year ago

It shouldn’t take that long. I think sometimes pip fails if you don’t have a c compiler set up and then the installation might not complete. That would be my best guess. Do you have clang installed on your system?

synapticlee commented 1 year ago

I have clang installed! Are there other package managers I could try besides npm and yarn?

zner0L commented 1 year ago

Well, we test against yarn, so if that fails there is bug we need to fix. Can you tell me the contents of your ~/.cache/autopy/venv directory?

synapticlee commented 1 year ago

It doesn't exist as I don't have autopy - do I need it?

baltpeter commented 1 year ago

Autopy should be installed automatically when you install appstraction. But @zner0L gave you the wrong path, on macOS, it's actually ~/Library/Caches/autopy/venv.

baltpeter commented 1 year ago

I'm currently testing this in a virtual machine with macOS Monterery 12.2.1 and Node 18.16.1 (I had to manually install Node 18 (brew install node@18)—frida-node doesn't work on newer versions currently). I'm seeing the same behaviour as you. npm i appstraction has now been running for almost half an hour and is still not done.

In the meantime, I've done some investigation. Here's what I found:

Autopy worked quickly and without issues. I have a Python installation in ~/Library/Caches/autopy/python/3.11.3 and a venv in ~/Library/Caches/autopy/venv/appstraction with the expected symlinks in ~/Library/Caches/autopy/venv/appstraction/bin.
Andromatic successfully created the licenses (~/Library/Caches/andromatic/licenses) and installed the Android command-line tools (~/Library/Caches/andromatic/cmdline-tools). That was also done within a few minutes.
Since then, the installation has been stuck. I can see a java process that has been at ~88% CPU usage the whole time. That process is essentially sdkmanager platform-tools build-tools;33.0.2 (the full command is a lot longer, but it's just the java command that the sdkmanager shell script expands into).
I'm not sure but I think that process is actually doing something. The CPU usage is fluctuating and the open files that Activity Monitor is showing are changing from time to time.

baltpeter commented 1 year ago

After 1:20h, it was still not finished and I gave up.

baltpeter commented 1 year ago

Turns out we already have an issue for this: https://github.com/tweaselORG/andromatic/issues/6

baltpeter commented 1 year ago

@synapticlee As a workaround until we have fixed and released this, you should be able to manually install Java, run /usr/libexec/java_home to find your Java home and export that as an env var (export JAVA_HOME=<path>).

synapticlee commented 1 year ago

Glad I wasn't the only one with this issue! I installed Java (I hadn't yet as this is a fairly new computer) and added it to the path. It is still hanging on the appstraction step though!

zner0L commented 1 year ago

Did you also export the JAVA_HOME? And do you have anything in the ~/Library/Caches/autopy/venv and ~/Library/Caches/andromatic folders?

synapticlee commented 1 year ago

I did export JAVA_HOME, yes, and in '~/Library/Caches/autopy/venv' I have appstraction and cyanoacrylate folders which are not empty (bin include lib pyvenv.cfg in cyanoacrylate, bin data include lib pyvenv.cfg share in appstraction). Nothing in ~/Library/Caches/andromatic

zner0L commented 1 year ago

Ok, I just tested it on my (yes, dated) MacBook Pro Mid 2015 with macOS Monterey 12.6.6. I installed appstraction with the following command:

$ JAVA_HOME=$(/usr/libexec/java_home) yarn add appstraction

It works, but it still takes quite a while (~ 109s). If that doesn’t work for you, maybe there is something off with your Java version? Mine is JDK 11:

$ /usr/libexec/java_home
/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home

baltpeter commented 1 year ago

I have just released a new version of andromatic that should fix this bug, so until we release the fix in appstraction, you can also try this:

# Install the newest version of andromatic manually first.
yarn add andromatic
# Then install appstraction. This should use the new, fixed version of andromatic instead of the old version that appstraction currently depends on.
yarn add appstraction

synapticlee commented 1 year ago

Re: the earlier conversation, I didn't have JDK installed, only Java, so I downloaded the latest JDK (20) and still having the same issue sadly. For the new fix, I successfully added andromatic but appstraction failed after that:

Error: ENOENT: no such file or directory, uv_cwd
    at process.wrappedCwd [as cwd] (node:internal/bootstrap/switches/does_own_process_state:126:28)
    at main (/usr/local/lib/node_modules/corepack/dist/corepack.js:16198:128)
    at Object.runMain (/usr/local/lib/node_modules/corepack/dist/corepack.js:16238:5)
    at Object.<anonymous> (/usr/local/lib/node_modules/corepack/dist/yarn.js:2:23)
    at Module._compile (node:internal/modules/cjs/loader:1218:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1272:10)
    at Module.load (node:internal/modules/cjs/loader:1081:32)
    at Module._load (node:internal/modules/cjs/loader:922:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47

synapticlee commented 1 year ago

Wait, nevermind, I tried it again and it worked!! Thanks! Now trying to install the tweasel-cli

synapticlee commented 1 year ago

Hmm sadly while installing yarn global add tweasel-cli, it's still hanging on the appstraction step for many many minutes, even though yarn add appstraction was successful, do you know why?

This is the output:

yarn global v1.22.19
warning ../../package.json: No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning "tweasel-cli > @oclif/core > ts-node@10.9.1" has unmet peer dependency "@types/node@*".
warning "tweasel-cli > @oclif/core > ts-node@10.9.1" has unmet peer dependency "typescript@>=2.7".
warning "tweasel-cli > cyanoacrylate > appstraction > ipa-extract-info > zipjs-browserify > browserify-shim@3.8.16" has unmet peer dependency "browserify@>= 2.3".
[4/4] 🔨  Building fresh packages...
[1/6] ⠁ frida
[2/6] ⠁ cpu-features
[-/6] ⠁ waiting...
[-/6] ⠁ waiting...
warning Error running install script for optional dependency: "/Users/jlee/.config/yarn/global/node_modules/cpu-features: Command failed.
Exit code: 127
Command: node buildcheck.js > buildcheck.gypi && node-gyp rebuild
Arguments
[5/6] ⠈ appstraction
[-/6] ⠁ waiting...
[-/6] ⠁ waiting...
[-/6] ⠁ waiting...
[-/6] ⠁ waiting...

baltpeter commented 1 year ago

Hmm sadly while installing yarn global add tweasel-cli, it's still hanging on the appstraction step for many many minutes, even though yarn add appstraction was successful, do you know why?

yarn global installs into a different node_modules directory than when you run yarn in a package, so yarn global add tweasel-cli won't use the updated andromatic that you installed elsewhere.

I haven't tested this but I would imagine that running yarn global add appstraction beforehand should fix this.

baltpeter commented 1 year ago

Error: ENOENT: no such file or directory, uv_cwd

I had a quick look at this to make sure it's not another bug we need to fix. As per a few reports of this error (https://github.com/joefitzgerald/go-plus/issues/249, https://stackoverflow.com/a/65127654, https://stackoverflow.com/a/68457191), this seems to happen if the directory you are in was deleted and should be fixed by cding to an existing directory and trying again. This probably explains why it worked on your second try.

synapticlee commented 1 year ago

Hmm sadly while installing yarn global add tweasel-cli, it's still hanging on the appstraction step for many many minutes, even though yarn add appstraction was successful, do you know why?

yarn global installs into a different node_modules directory than when you run yarn in a package, so yarn global add tweasel-cli won't use the updated andromatic that you installed elsewhere.

I haven't tested this but I would imagine that running yarn global add appstraction beforehand should fix this.

Thanks, this makes sense! Running yarn global add andromatic worked but weirdly yarn global add appstraction is hanging, even though installing it locally worked earlier..

zner0L commented 1 year ago

I am thinking, since you already tried to install appstraction in the global context that might have broken the dependencies. Try to remove both and install them in the correct order (I just checked that it works that way on my Macbook):

$ yarn global remove andromatic appstraction
$ yarn global add andromatic
$ yarn global add appstraction

baltpeter commented 1 year ago

@zner0L I'm assuming the latter two were also actually yarn global?

synapticlee commented 1 year ago

After fiddling around a little I was able to download tweasel-cli! Thanks everyone for your help. Hopefully that's the end of the issues

baltpeter commented 1 year ago

Glad to hear that! Thanks for helping us debug.

tweaselORG / appstraction

Appstraction download hangs and does not complete #108