search

aws / aws-toolkit-jetbrains

AWS Toolkit for JetBrains - a plugin for interacting with AWS from JetBrains IDEs
https://plugins.jetbrains.com/plugin/11349-aws-toolkit
Apache License 2.0
732 stars 200 forks source link

PyCharm - AWS Plugin does not show S3 buckets #2174

Closed armaciej closed 3 years ago

armaciej commented 3 years ago

Describe the bug AWS Plugin for PyCharm does not show S3 buckets, but produces an error:

2020-10-28 21:12:55,101 [ 191846] INFO - ains.services.s3.S3ServiceNode - Failed to get children java.util.concurrent.TimeoutException at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) at software.aws.toolkits.jetbrains.core.AwsResourceCache$Companion.wait(AwsResourceCache.kt:132) at software.aws.toolkits.jetbrains.core.AwsResourceCache$Companion.access$wait(AwsResourceCache.kt:127) at software.aws.toolkits.jetbrains.core.AwsResourceCache$DefaultImpls.getResourceNow(AwsResourceCache.kt:85) at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache.getResourceNow(AwsResourceCache.kt:260) at software.aws.toolkits.jetbrains.core.AwsResourceCache$DefaultImpls.getResourceNow(AwsResourceCache.kt:96) at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache.getResourceNow(AwsResourceCache.kt:260) at software.aws.toolkits.jetbrains.core.AwsResourceCacheKt.getResourceNow(AwsResourceCache.kt:156) at software.aws.toolkits.jetbrains.core.AwsResourceCacheKt.getResourceNow$default(AwsResourceCache.kt:155) at software.aws.toolkits.jetbrains.services.s3.S3ServiceNode.getChildrenInternal(S3ExplorerNode.kt:20) at software.aws.toolkits.jetbrains.core.explorer.nodes.ResourceParentNode$DefaultImpls.getChildren(ResourceParentNode.kt:20) at software.aws.toolkits.jetbrains.core.explorer.nodes.AwsExplorerServiceRootNode.getChildren(AwsExplorerServiceRootNode.kt:13) at software.aws.toolkits.jetbrains.core.explorer.nodes.AwsExplorerServiceRootNode.getChildren(AwsExplorerServiceRootNode.kt:13) at com.intellij.ide.util.treeView.AbstractTreeStructureBase.getChildElements(AbstractTreeStructureBase.java:34) at software.aws.toolkits.jetbrains.ui.tree.StructureTreeModel.getValidChildren(StructureTreeModel.java:414) at software.aws.toolkits.jetbrains.ui.tree.StructureTreeModel.validateChildren(StructureTreeModel.java:332) at software.aws.toolkits.jetbrains.ui.tree.StructureTreeModel.getNode(StructureTreeModel.java:326) at software.aws.toolkits.jetbrains.ui.tree.StructureTreeModel.getChildren(StructureTreeModel.java:346) at software.aws.toolkits.jetbrains.ui.tree.AsyncTreeModel$CmdGetChildren.getNode(AsyncTreeModel.java:582) at software.aws.toolkits.jetbrains.ui.tree.AsyncTreeModel$ObsolescentCommand.get(AsyncTreeModel.java:476) at software.aws.toolkits.jetbrains.ui.tree.AsyncTreeModel$ObsolescentCommand.get(AsyncTreeModel.java:442) at software.aws.toolkits.jetbrains.ui.tree.Command$Processor.lambda$process$1(Command.java:51) at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:170) at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:629) at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:581) at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:60) at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:157) at software.aws.toolkits.jetbrains.ui.tree.Invoker.invokeSafely(Invoker.java:147) at software.aws.toolkits.jetbrains.ui.tree.Invoker.lambda$invokeLater$0(Invoker.java:93) at software.aws.toolkits.jetbrains.ui.tree.Invoker$BackgroundThread.lambda$offer$0(Invoker.java:306) at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:215) at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:26) at com.intellij.util.concurrency.BoundedTaskExecutor$1.execute(BoundedTaskExecutor.java:194) at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:207) at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:183) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668) at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665) at java.base/java.lang.Thread.run(Thread.java:834)

On the frontend: image

To reproduce Not sure

Expected behavior The plugin should show the list of S3 buckets.

Your Environment

abrooksv commented 3 years ago

We have the default timeout for all resources set to 30secs, s3 though requires a lot of AWS calls that happen sequentially to do the filtering.

Things we can look into:

  1. We could increase the timeout for the S3 explorer node since we know that call is expensive.
  2. Create a parallel Resource.view to do something like:
runBlocking {
    map { async(IO) { bucketRegionLookup(it) } }.map { it.await() }
}

If this is blocking you, you can work around it by forcing the S3 buckets to load in a different way than the explorer tree that isn't bound by max timeout: Such as going to Lambda->Create Lambda which will force the S3 buckets to load and be cached for that region.

armaciej commented 3 years ago

Is there a way to pull out timeouts to settings? Maybe an idea for a feature?

Pozdrawiam, Arek

pt., 30 paź 2020, 02:30 użytkownik Austin Brooks notifications@github.com napisał:

We have the default timeout for all resources set to 30secs, s3 though requires a lot of AWS calls that happen sequentially to do the filtering.

Things we can look into:

  1. We could increase the timeout for the S3 explorer node since we know that call is expensive.
  2. Create a parallel Resource.view to do something like:

runBlocking { map { async(IO) { bucketRegionLookup(it) } }.map { it.await() } }

If this is blocking you, you can work around it by forcing the S3 buckets to load in a different way than the explorer tree that isn't bound by max timeout: Such as going to Lambda->Create Lambda which will force the S3 buckets to load and be cached for that region.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws/aws-toolkit-jetbrains/issues/2174#issuecomment-719115802, or unsubscribe https://github.com/notifications/unsubscribe-auth/AED6XFUND6AELQI7CHXB6O3SNIJMBANCNFSM4TC2LVJA .

abrooksv commented 3 years ago

We can do it in the hidden IDE settings registry, which is where we keep the timeouts for credential_process. Putting it normal Settings menu might not make sense since it's a default and not global so not sure how we would communicate that to users.

It impacts how we coded the toolkit and if that particular function call uses the default or not. i.e. if they set it to say 60secs, but the function call isn't using the default, the setting would have no effect.

abrooksv commented 3 years ago

@armaciej Roughly how many buckets do you have?

hunterwerlla commented 3 years ago

@armaciej I was able to repro this with a slow connection. I merged a fix that will be in the next release that speeds up loading buckets significantly and should fix your issue.

armaciej commented 3 years ago

I have around 90 buckets.

Pozdrawiam, Arek

pt., 6 lis 2020, 20:23 użytkownik Hunter Werlla notifications@github.com napisał:

@armaciej https://github.com/armaciej I was able to repro this with a slow connection. I merged a fix that will be in the next release that speeds up loading buckets significantly and should fix your issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws/aws-toolkit-jetbrains/issues/2174#issuecomment-723256579, or unsubscribe https://github.com/notifications/unsubscribe-auth/AED6XFUU6KCJCZSESCQZODTSOREL5ANCNFSM4TC2LVJA .

abrooksv commented 3 years ago

We have sped up the loading of S3 buckets in 1.21

armaciej commented 3 years ago

Thanks - it looks like it's working for me now! Great job:)

On Tue, Nov 24, 2020 at 6:50 PM Austin Brooks notifications@github.com wrote:

We have sped up the loading of S3 buckets in 1.21

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws/aws-toolkit-jetbrains/issues/2174#issuecomment-733137926, or unsubscribe https://github.com/notifications/unsubscribe-auth/AED6XFU6RE5ZBSTOGDDQMZLSRPW7PANCNFSM4TC2LVJA .

-- Kind regards, Arek