bower / bower

A package manager for the web
bower.io
MIT License
14.99k stars 1.85k forks source link

ENOENT after 1.2.8 update #991

Closed wilkerlucio closed 10 years ago

wilkerlucio commented 10 years ago

hi, since the last update I'm getting this error when trying to install my packages:

bower chosen.jquery#*           cached https://github.com/harvesthq/chosen/releases/download/1.0.0/chosen_v1.0.0.zip#e-tag:3faf39b4e
bower chosen.jquery#*         validate e-tag:3faf39b4e against https://github.com/harvesthq/chosen/releases/download/1.0.0/chosen_v1.0.0.zip#*
bower angular#v1.2.0-rc.2       cached git://github.com/angular/bower-angular.git#1.2.0-rc.2
bower angular#v1.2.0-rc.2     validate 1.2.0-rc.2 against git://github.com/angular/bower-angular.git#v1.2.0-rc.2
bower jquery#~2.0.3             cached git://github.com/components/jquery.git#2.0.3
bower jquery#~2.0.3           validate 2.0.3 against git://github.com/components/jquery.git#~2.0.3
bower bootstrap#~3.0.0          cached git://github.com/twbs/bootstrap.git#3.0.2
bower bootstrap#~3.0.0        validate 3.0.2 against git://github.com/twbs/bootstrap.git#~3.0.0
bower jquery#>= 1.9.0           cached git://github.com/components/jquery.git#2.0.3
bower jquery#>= 1.9.0         validate 2.0.3 against git://github.com/components/jquery.git#>= 1.9.0
bower chosen.jquery#*              new version for https://github.com/harvesthq/chosen/releases/download/1.0.0/chosen_v1.0.0.zip#*
bower chosen.jquery#*          resolve https://github.com/harvesthq/chosen/releases/download/1.0.0/chosen_v1.0.0.zip#*
bower chosen.jquery#*         download https://github.com/harvesthq/chosen/releases/download/1.0.0/chosen_v1.0.0.zip
bower chosen.jquery#*          extract chosen_v1.0.0.zip
bower chosen.jquery#*           ENOENT ENOENT, open '/var/folders/nb/m18105410nz_xkx4qkd98h_c0000gn/T/wilkerlucio/bower/chosen.jquery-46279-SZy2v5/docsupport/'

Stack trace:
Error: ENOENT, open '/var/folders/nb/m18105410nz_xkx4qkd98h_c0000gn/T/wilkerlucio/bower/chosen.jquery-46279-SZy2v5/docsupport/'

Console trace:
Trace
    at StandardRenderer.error (/usr/local/lib/node_modules/bower/lib/renderers/StandardRenderer.js:74:17)
    at Logger.updateNotifier.packageName (/usr/local/lib/node_modules/bower/bin/bower:109:18)
    at Logger.EventEmitter.emit (events.js:95:17)
    at Logger.emit (/usr/local/lib/node_modules/bower/node_modules/bower-logger/lib/Logger.js:29:39)
    at /usr/local/lib/node_modules/bower/lib/commands/install.js:27:16
    at _rejected (/usr/local/lib/node_modules/bower/node_modules/q/q.js:808:24)
    at /usr/local/lib/node_modules/bower/node_modules/q/q.js:834:30
    at Promise.when (/usr/local/lib/node_modules/bower/node_modules/q/q.js:1079:31)
    at Promise.promise.promiseDispatch (/usr/local/lib/node_modules/bower/node_modules/q/q.js:752:41)
    at /usr/local/lib/node_modules/bower/node_modules/q/q.js:574:44

System info:
Bower version: 1.2.8
Node version: 0.10.21
OS: Darwin 13.0.0 x64
wilkerlucio commented 10 years ago

maybe something related to #988 ?

wibblymat commented 10 years ago

It's decompress-zip related. The variety of zip files around amazes me. This one ignores the fields that are supposed to tell you if an entry is a directory. Instead, the directory has all zeroes in those fields. Which would make it look exactly like an empty file (which I have also seen) except that the name ends in a '/'.

sindresorhus commented 10 years ago

@wibblymat lol, sounds fun...

danielepolencic commented 10 years ago

Same here. Bower fails on Heroku with Angular 1.2.4 with the same error (works with 1.2.3, though). Any workaround?

danielmcormond commented 10 years ago

@danielepolencic For now, try downgrading to Bower 1.2.7, that worked for me.

astral303 commented 10 years ago

+1 same problem with chosen

jhsimms commented 10 years ago

Perhaps this is related to #933. We're currently experiencing this in Jenkins, with about 2/3 of our builds failing because of it.

flashjames commented 10 years ago

I have the same error on Teamcity CI, but for angular.js package:

[15:34:19][Step 1/3] bower angular#>=1                                         ENOTEMPTY ENOTEMPTY, rename '.bower-tmp/angular-24804-3AHPOf'
[15:34:19][Step 1/3] 
[15:34:19][Step 1/3] Stack trace:
[15:34:19][Step 1/3] Error: ENOTEMPTY, rename '.bower-tmp/angular-24804-3AHPOf'
[15:34:19][Step 1/3] 
[15:34:19][Step 1/3] Console trace:
[15:34:19][Step 1/3] Trace
[15:34:19][Step 1/3]     at StandardRenderer.error   (/home/builds/TeamCity/buildAgent/work/f26358856765a6f2/node_modules/bower/lib/renderers/StandardRenderer.js:74:17)
[15:34:19][Step 1/3]     at Logger.<anonymous> (/home/builds/TeamCity/buildAgent/work/f26358856765a6f2/node_modules/bower/bin/bower:109:18)
[15:34:19][Step 1/3]     at Logger.EventEmitter.emit (events.js:88:17)
[15:34:19][Step 1/3]     at Logger.emit (/home/builds/TeamCity/buildAgent/work/f26358856765a6f2/node_modules/bower/node_modules/bower-logger/lib/Logger.js:29:39)
[15:34:19][Step 1/3]     at /home/builds/TeamCity/buildAgent/work/f26358856765a6f2/node_modules/bower/lib/commands/install.js:27:16
[15:34:19][Step 1/3]     at _rejected (/home/builds/TeamCity/buildAgent/work/f26358856765a6f2/node_modules/bower/node_modules/q/q.js:808:24)
[15:34:19][Step 1/3]     at Promise.then.self.promiseDispatch.threw (/home/builds/TeamCity/buildAgent/work/f26358856765a6f2/node_modules/bower/node_modules/q/q.js:834:30)
[15:34:19][Step 1/3]     at Promise.when (/home/builds/TeamCity/buildAgent/work/f26358856765a6f2/node_modules/bower/node_modules/q/q.js:1079:31)
[15:34:19][Step 1/3]     at Promise.promise.promiseDispatch (/home/builds/TeamCity/buildAgent/work/f26358856765a6f2/node_modules/bower/node_modules/q/q.js:752:41)
[15:34:19][Step 1/3]     at messages (/home/builds/TeamCity/buildAgent/work/f26358856765a6f2/node_modules/bower/node_modules/q/q.js:574:44)
ztbrown commented 10 years ago

Same problem here (CircleCI). Having this issue with 1.2.6, 1.2.7 and 1.2.8:

Stack trace: Error: ENOTEMPTY, rename '.bower-tmp/angular-10069-HGrInP'

Console trace: Trace at StandardRenderer.error (/home/ubuntu/nvm/v0.8.12/lib/node_modules/bower/lib/renderers/StandardRenderer.js:74:17) at Logger. (/home/ubuntu/nvm/v0.8.12/lib/node_modules/bower/bin/bower:109:18) at Logger.EventEmitter.emit (events.js:93:17) at Logger.emit (/home/ubuntu/nvm/v0.8.12/lib/node_modules/bower/node_modules/bower-logger/lib/Logger.js:29:39) at /home/ubuntu/nvm/v0.8.12/lib/node_modules/bower/lib/commands/install.js:28:16 at _rejected (/home/ubuntu/nvm/v0.8.12/lib/node_modules/bower/node_modules/q/q.js:808:24) at Promise.then.self.promiseDispatch.threw (/home/ubuntu/nvm/v0.8.12/lib/node_modules/bower/node_modules/q/q.js:834:30) at Promise.when (/home/ubuntu/nvm/v0.8.12/lib/node_modules/bower/node_modules/q/q.js:1079:31) at Promise.promise.promiseDispatch (/home/ubuntu/nvm/v0.8.12/lib/node_modules/bower/node_modules/q/q.js:752:41) at messages (/home/ubuntu/nvm/v0.8.12/lib/node_modules/bower/node_modules/q/q.js:574:44)

System info: Bower version: 1.2.6 Node version: 0.8.12 OS: Linux 3.8.0-32-generic x64

diwu1989 commented 10 years ago

Really really sad that I have to stick with Bower 1.2.5 just to avoid flakiness.

svemir commented 10 years ago

For me this seems to happen only when there is some sort of a network lag or some other external factor. The same operation (bower update) always works on one box, but on another box it sometimes fails and sometimes works. Both boxes have bower 1.2.8 and identical bower.json file. The two boxes are on different networks, and the one that fails is very busy and does have occasional network issues.

ztbrown commented 10 years ago

I get the same flakiness on 1.2.5

wibblymat commented 10 years ago

I think that the comments here are talking about two or more issues. Any issue with bower < 1.2.8 is definitely not the same as the original report.

wibblymat commented 10 years ago

Sorry, didn't meant to close. Need to do a new release of decompress-zip and then bump the dependency version in bower.

wibblymat commented 10 years ago

I've released a new version of decompress-zip. If you uninstall and reinstall bower (not just npm update) then it should use the new one.

@sindresorhus @paulirish @satazor should I bump the dependency version too?

sindresorhus commented 10 years ago

@wibblymat no need, it's using ~.

danielmcormond commented 10 years ago

@wibblymat Thanks! It's working well for me here. :+1:

adrianboimvaser commented 10 years ago

Please, help!

I can't get past

Error: ENOTEMPTY, rename '.bower-tmp/angular-9388-40CW6C'

Thanks.

otaviosoares commented 10 years ago

Same error to install angular

Stack trace: Error: ENOTEMPTY, rename '/tmp/ubuntu/bower/angular-10439-QWqdct'

Console trace: Trace at StandardRenderer.error (/usr/lib/node_modules/bower/lib/renderers/StandardRenderer.js:74:17) at Logger.updateNotifier.packageName (/usr/lib/node_modules/bower/bin/bower:109:18) at Logger.EventEmitter.emit (events.js:95:17) at Logger.emit (/usr/lib/node_modules/bower/node_modules/bower-logger/lib/Logger.js:29:39) at /usr/lib/node_modules/bower/lib/commands/install.js:27:16 at _rejected (/usr/lib/node_modules/bower/node_modules/q/q.js:808:24) at /usr/lib/node_modules/bower/node_modules/q/q.js:834:30 at Promise.when (/usr/lib/node_modules/bower/node_modules/q/q.js:1079:31) at Promise.promise.promiseDispatch (/usr/lib/node_modules/bower/node_modules/q/q.js:752:41) at /usr/lib/node_modules/bower/node_modules/q/q.js:574:44

danielmcormond commented 10 years ago

@adrianboimvaser @otaviosoares Did you follow Mat's instructions?

npm uninstall -g bower && npm install -g bower

I would also recommend cleaning your cache before trying your bower install:

bower cache clean

Let us know if this helps.

otaviosoares commented 10 years ago

I did, but I got the same error.

Thank you.

adrianboimvaser commented 10 years ago

Thank you, but that didn't help at all.

wibblymat commented 10 years ago

@otaviosoares @adrianboimvaser it would be great if you could provide a little more information. Can you let me know which OS you are on?

If you change to a new directory outside of your project and bower cache clean && bower install angular do you get the same issue?

wibblymat commented 10 years ago

Node version, too, actually.

knalli commented 10 years ago

Well, I have a ci system (Jenkins, RHEL 5, Linux 2.6.32-131.6.1.el6.x86_64 x64) with NodeJS 0.10.22 (via nvm in user dir), Npm 1.3.14, Bower 1.2.7 (fixed manually to avoid this issue here); the build script uses explicitly following lines

# Execute
node_modules/.bin/bower cache clean
node_modules/.bin/bower install

And yes, the internet connection is sometimes a little bit slow. A slow download rate or even a retry is possible due the connection itself or even Github itself.

There are two errors in the log

ower angular#1.0.8                                                            retry Download of https://github.com/angular/bower-angular/archive/v1.0.8.tar.gz failed with ETIMEDOUT, retrying in 1.0s
bower json3#~3.2.5                                                          progress received 2.9MB of 4.9MB downloaded, 60%
bower angular#~1.x                                                             retry Download of https://github.com/angular/bower-angular/archive/v1.2.9-build.2122+sha.34fee06.tar.gz failed with ETIMEDOUT, retrying in 1.5s
bower angular#>= 1.0.2                                                         retry Download of https://github.com/angular/bower-angular/archive/v1.2.9-build.2122+sha.34fee06.tar.gz failed with ETIMEDOUT, retrying in 1.7s
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at CleartextStream.EventEmitter.addListener (events.js:160:15)
    at CleartextStream.Readable.on (_stream_readable.js:689:33)
    at CleartextStream.EventEmitter.once (events.js:179:8)
    at Request.onResponse (/PROJECTDIR/node_modules/bower/node_modules/request/request.js:625:25)
    at ClientRequest.g (events.js:175:14)
    at ClientRequest.EventEmitter.emit (events.js:95:17)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1688:21)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:121:23)
    at CleartextStream.socketOnData [as ondata] (http.js:1583:20)
    at CleartextStream.read [as _read] (tls.js:507:12)

(=> sounds like an uncaught exception pattern)

and

bower angular#>= 1.0.2                                                      resolved git://github.com/angular/bower-angular.git#1.2.9-build.2122+sha.34fee06
bower angular#~1.2.0                                                       ENOTEMPTY ENOTEMPTY, rename '.bower-tmp/angular-11823-QpFFs6'

Stack trace:
Error: ENOTEMPTY, rename '.bower-tmp/angular-11823-QpFFs6'

Console trace:
Trace
    at StandardRenderer.error (/PROJECTDIR/node_modules/bower/lib/renderers/StandardRenderer.js:74:17)
    at Logger.updateNotifier.packageName (/PROJECTDIR/node_modules/bower/bin/bower:109:18)
    at Logger.EventEmitter.emit (events.js:95:17)
    at Logger.emit (/PROJECTDIR/node_modules/bower/node_modules/bower-logger/lib/Logger.js:29:39)
    at /PROJECTDIR/node_modules/bower/lib/commands/update.js:24:16
    at _rejected (/PROJECTDIR/node_modules/bower/node_modules/q/q.js:808:24)
    at /PROJECTDIR/node_modules/bower/node_modules/q/q.js:834:30
    at Promise.when (/PROJECTDIR/node_modules/bower/node_modules/q/q.js:1079:31)
    at Promise.promise.promiseDispatch (/PROJECTDIR/node_modules/bower/node_modules/q/q.js:752:41)
    at /PROJECTDIR/node_modules/bower/node_modules/q/q.js:574:44

(possibly an impact of the error before?)

This error occurs only sometimes, and is mostly unique (next time probably another package).

otaviosoares commented 10 years ago

Ubuntu 12.04 Node 0.10.20 Bower 1.2.8

If I try to install angular outside my project dir it works.

Hope it helps.

Thanks

tfnico commented 10 years ago

Re-installing 1.2.8 worked for me :+1: Thanks!

tfmorris commented 10 years ago

So the difference between the broken version and the fixed version is that one is 1.2.8 and the other is 1.2.8? That doesn't sound very obvious.

Since decompress-zip 0.0.3 is known to be broken in a way that affects bower badly, shouldn't the dependencies be updated to reflect this?

tfmorris commented 10 years ago

I've created a pull request #1051 with the necessary changes.

tfmorris commented 10 years ago

@paulirish @satazor @wibblymat We appear to have one vote for maintaining the status quo of not being able to distinguish working from broken versions of Bower. What say the rest of you?

The current decompress-zip dependency is ~0.0.3 which means that 0.0.3 is an acceptable version, when we all know it isn't. Bumping both the dependency version re-releasing Bower will mean that engineers depending on this tool will stop wasting time debugging a problem which has already been fixed.

jacqueslareau commented 10 years ago

re-installing 1.2.8 worked for me too. really not obvious.

ztbrown commented 10 years ago

:+1: to @tfmorris

diwu1989 commented 10 years ago

Definitely vote to version bump the dependency, version bump bower, and then everyone ends up with a guaranteed working version of Bower.

@tfmorris I saw that @sindresorhus decided to close your PR, and I don't think it should have been closed.

Bower itself is broken because it incorrectly depended on a buggy version of decompress-zip, and I think to the end user, even though the bug is in decompress-zip, it is a valid Bower bug that needs to be fixed by bumping the dependency version.

Closing your PR is the same as just shifting the blame over to someone else, and it doesn't make Bower less buggy.

sindresorhus commented 10 years ago

This is how npm works, you're barking up the wrong tree here. All npm modules does this. npm modules use tilde versions at the benefit of getting bugfixes from sub-dependencies without having to release a new version. Everyone installing Bower now get's the right version. If you're hitting an error you should be reinstalling anyways.

@wibblymat is free to do a release if he wants to do so. I just don't see the point.

Aethon commented 10 years ago

http://semver.org/spec/v2.0.0.html

"3. Once a versioned package has been released, the contents of that version MUST NOT be modified. Any modifications MUST be released as a new version."

All my votes to following this rule.

chmontgomery commented 10 years ago

agree with @Aethon. We are just inviting unexpected breaks like this using ~. As well intentioned as minor releases are, sometimes they introduce bugs rather than fix them (as seen here). The only way to guarantee bower will continue to run the same way no matter when you download it is to have fixed versions.

jacqueslareau commented 10 years ago

Why is this issue closed? Bower is not working for me anymore. Deinstalled and reinstalled everything.

nullivex commented 10 years ago

Confirmed not working on two different computers.

bower angular#1.2.11-build.2197+sha.c22ab5d            EPERM EPERM, rename 'C:\Users\JT4\AppData\Roaming\bower\cache\packages\ef2188def21eb1bbd1f1792311942a53\1.2.11-build.2197%2Bsha.
c22ab5d'

Stack trace:
Error: EPERM, rename 'C:\Users\JT4\AppData\Roaming\bower\cache\packages\ef2188def21eb1bbd1f1792311942a53\1.2.11-build.2197%2Bsha.c22ab5d'

Console trace:
Trace
    at StandardRenderer.error (C:\Users\JT4\nodist\bin\node_modules\bower\lib\renderers\StandardRenderer.js:74:17)
    at Logger.updateNotifier.packageName (C:\Users\JT4\nodist\bin\node_modules\bower\bin\bower:109:18)
    at Logger.EventEmitter.emit (events.js:95:17)
    at Logger.emit (C:\Users\JT4\nodist\bin\node_modules\bower\node_modules\bower-logger\lib\Logger.js:29:39)
    at C:\Users\JT4\nodist\bin\node_modules\bower\lib\commands\install.js:27:16
    at _rejected (C:\Users\JT4\nodist\bin\node_modules\bower\node_modules\q\q.js:808:24)
    at C:\Users\JT4\nodist\bin\node_modules\bower\node_modules\q\q.js:834:30
    at Promise.when (C:\Users\JT4\nodist\bin\node_modules\bower\node_modules\q\q.js:1079:31)
    at Promise.promise.promiseDispatch (C:\Users\JT4\nodist\bin\node_modules\bower\node_modules\q\q.js:752:41)
    at C:\Users\JT4\nodist\bin\node_modules\bower\node_modules\q\q.js:574:44

System info:
Bower version: 1.2.8
Node version: 0.10.24
OS: Windows_NT 6.1.7601 ia32

Tried reinstalling bower many times. Tried administrator and normal user.

jahkeup commented 10 years ago

Yep still an issue. Followed the advice on this and https://github.com/bower/bower/issues/933 and nada.

bower angular#>=1.2.0                               resolved git://github.com/angular/bower-angular.git#1.2.13-build.2242+sha.e645f7c
bower angular#1.2.13-build.2242+sha.e645f7c         resolved git://github.com/angular/bower-angular.git#1.2.13-build.2242+sha.e645f7c
bower angular#>=1.0.0 <1.3.0                       ENOTEMPTY ENOTEMPTY, rename '/home/exzigo/.cache/bower/packages/ef2188def21eb1bbd1f1792311942a53/1.2.13-build.2242%2Bsha.e645f7c'

Stack trace:
Error: ENOTEMPTY, rename '/home/exzigo/.cache/bower/packages/ef2188def21eb1bbd1f1792311942a53/1.2.13-build.2242%2Bsha.e645f7c'

Console trace:
Trace
    at StandardRenderer.error (/usr/local/lib/node_modules/bower/lib/renderers/StandardRenderer.js:74:17)
    at Logger.updateNotifier.packageName (/usr/local/lib/node_modules/bower/bin/bower:109:18)
    at Logger.EventEmitter.emit (events.js:95:17)
    at Logger.emit (/usr/local/lib/node_modules/bower/node_modules/bower-logger/lib/Logger.js:29:39)
    at /usr/local/lib/node_modules/bower/lib/commands/install.js:27:16
    at _rejected (/usr/local/lib/node_modules/bower/node_modules/q/q.js:808:24)
    at /usr/local/lib/node_modules/bower/node_modules/q/q.js:834:30
    at Promise.when (/usr/local/lib/node_modules/bower/node_modules/q/q.js:1079:31)
    at Promise.promise.promiseDispatch (/usr/local/lib/node_modules/bower/node_modules/q/q.js:752:41)
    at /usr/local/lib/node_modules/bower/node_modules/q/q.js:574:44

System info:
Bower version: 1.2.8
Node version: 0.10.25
OS: Linux 3.2.0-4-amd64 x64

And I'm getting this error for random packages, it has been "preferring" Angular though.

AnthonySteele commented 10 years ago

This seems to work for me now to fix the issue:

bower cache clean
npm uninstall -g bower
npm install -g bower
npm update decompress-zip

Edit: Nope, not a reliable fix.

yjaaidi commented 10 years ago

This solved it for me:

In ".bowerrc":

{
    "storage": {
        "packages": ".bower-cache",
        "registry": ".bower-registry"
    },
    "tmp": ".bower-tmp"
}
timelf123 commented 10 years ago

@yjaaidi that fixed it for me too, thanks!

yjaaidi commented 10 years ago

@timelf123 glad that it helped you too!

knalli commented 10 years ago

This is still not solved, nothing had changed.

Hm, well. Because we already know the issue, we are also knowing this solution actually does not solve the problem. It is only luck. It happened more or less randomly.

So: What are the plans about bower? Why will be this issue not tracked (obviously)? Why is there still no solution as a release? What's the hitch? Can we help? How?

I can understand my patch was not the right way.. but I cannot understand why we are still talking about this here. Anyone with a CI environment is probably faced with that problems occasional (better: haphazardly) because of the nature of the issue.

tfnico commented 10 years ago

This problem reminds me of using SNAPSHOT dependencies in Maven. It's a great to have these "floating" dependency version declarations when you are waiting for other teams to deliver frequent fixes on their libraries you depend on in your application.

But you can't release a project before you have resolved all the SNAPSHOT versions.

Maven also supports version ranges, which do look more like the npm tilde mechanism. But version ranges is a mechanism made to help ease resolving conflicting dependency versions between transitive dependencies (i.e. libraries depending on other libraries). You'll never have version ranges in your end-product application - there won't be any other applications to conflict with, so you can allow more strictness, so to speak.

Maybe the parallel is not well drawn into the npm world, but Maven's design choices and versioning habits were developed through years of people running into problems just like this. Having floating dependency versions in released versions of an end product (like bower) is a bad idea.

ilanbiala commented 10 years ago

I had this issue too, but I fixed it by running npm install bower -g --no-cache and then running the same in my project folder without the -g (npm install bower --no-cache).

theoreticaLee commented 10 years ago

Bump, having the same issue after trying all the above solutions.

knalli commented 10 years ago

Nope, that won't fix the issue. NPM is not the problem. Neither an activated nor an inactive cache will solve the issue. Sorry.

cuongdd2 commented 10 years ago

I got the same problem when installing http://jqueryui.com/resources/download/jquery-ui-themes-1.10.4.zip

System info:
decompress-zip 0.0.4
Bower version: 1.2.8
Node version: 0.10.25
OS: Windows_NT 6.1.7601 x64
jhicken commented 10 years ago

The solution that worked for us on our Jenkins CI integration was to completely remove the bower_components or equivalent if you have a different one defined in your '.bowerrc'. I know this is not a permeant solution but if you need a consistent fix for CI... This is a temporary solution.