aws / studio-lab-examples

Example notebooks for working with SageMaker Studio Lab. Sign up for an account at the link below!
https://studiolab.sagemaker.aws
Apache License 2.0
624 stars 181 forks source link

There is no runtime available right now. Please change the compute type or try again later. #110

Open kanlicali opened 2 years ago

kanlicali commented 2 years ago

this message again again again. i am not use gpu

There is no runtime available right now. Please change the compute type or try again later.

there is a problem.

In previous posts, it was written to delete the account and sign up again.

i did but still the same

icoxfog417 commented 2 years ago

@kanlicali , sorry for unavailability of of GPU. Could you please upvote (👍 ) the following issue instead raising the new issue?

https://github.com/aws/studio-lab-examples/issues/98

We can recognize the unavailability quantitatively. If you have the other acceptable criteria described on this issue (in #98, low spec GPU is acceptable than unavailable), let me know us.

kanlicali commented 2 years ago

gpu won't turn on at all. it keeps giving error. I've been trying for 1 hour. I pressed it maybe 1000 times. but always the same message: There is no runtime available right now. Please change the compute type or try again later

It's not a performance vs warning as it says there.

System won't start when I select gpu.

kanlicali commented 2 years ago

https://imgyukle.com/i/RFXCW0

kanlicali commented 2 years ago

@icoxfog417 https://imgyukle.com/i/RFXCW0

MicheleMonclova commented 2 years ago

Thank you Kanlicali for your message. we are experiencing a high demand for gpu which we are working on. In the meantime can you tell me why CPU won't suffice? If we can understand your use case it could help us understand better how much GPU we need to make available.

kanlicali commented 2 years ago

I understand there is a lot of demand.

but only 1 week. no gpu turned on

Like I said, I pressed it maybe 1000 times. result no gpu.

it doesn't come across

datanetai commented 2 years ago

Thank you Kanlicali for your message. we are experiencing a high demand for gpu which we are working on. In the meantime can you tell me why CPU won't suffice? If we can understand your use case it could help us understand better how much GPU we need to make available.

Hi I'm recently using sagemaker lab for time series analysis. CPU work best for that. But i'm also learning some deep learning which i want to experiment on. But only one time i was able to get GPU for last weeks. Which is fine due to high demand for gpu but i suggest to put daily limit on gpu usage for free service. I also like the solution presented in #98

MicheleMonclova commented 2 years ago

Thanks Mohammadameerhaza, we are working on a few things to alleviate this problem. Stay tuned!

Faith-Uchiha commented 2 years ago

I met the same question ! i can use cpu runtime but gpu is always unavailable

icoxfog417 commented 2 years ago

Dear @kanlicali, We improved GPU allocation recently. We may launch GPU instance by 1 trial. Could you please try launching GPU instance? If the problem solved, please let us know by closing this issue.

Faith-Uchiha commented 2 years ago

这是来自QQ邮箱的自动回复邮件。   您好,您给我发送的邮件已收到。

RaistlinD2x commented 1 year ago

Thank you Kanlicali for your message. we are experiencing a high demand for gpu which we are working on. In the meantime can you tell me why CPU won't suffice? If we can understand your use case it could help us understand better how much GPU we need to make available.

Anyone building a deep learning model will generally want to use a GPU. Basic stats from some of my tests show a 30x speed increase. For a 'small' training run that can be the difference between 1 hour and 30 hours. I'm also pretty sure the ONLY reason anyone uses a cloud based jupyter environment is to gain access to GPU instances, any modern PC can handle basic CPU efforts. If I'm wrong about my assumptions then please let me know but that is really the only reason I can see someone doing this so they can avoid a 2-3000$ spend on a GPU worth using.

Just for my specific needs, I intentionally migrated to Sagemaker Studio Labs specifically to avoid this issue. Colab from Google offers no guarantee on compute for GPU time even for the $30/month paid sub. Studio Labs states without ambiguity that we will have access to a GPU for 8 hours per day, the fact that this is happening is against an implicit SLA. I would highly suggest you change the wording if you want people to assume they may not receive what you're saying they can have.

RaistlinD2x commented 1 year ago

Thanks Mohammadameerhaza, we are working on a few things to alleviate this problem. Stay tuned!

This message is now over half a year old. Any update?

RaistlinD2x commented 1 year ago

And a final comment, now that I've given up on requesting a GPU I have attempted to restart my session with a CPU. I am now receiving the message "The server has received too many requests."

JannicCutura commented 1 year ago

can neither start CPU nor GPU right now. With GPU I understand, they are expensive and it is rather generous that you offer them for free. But with CPU many other providers reliably make it available...

Faith-Uchiha commented 1 year ago

这是来自QQ邮箱的自动回复邮件。   您好,您给我发送的邮件已收到。

MicheleMonclova commented 1 year ago

Hi All, we are experiencing elevated error starting runtimes for GPU. The SageMaker Studio Lab team is working to restore the service.Please use CPU runtime until we are fully recovered. We apologize for any inconvenience.

The FAQ does state the upper limits for GPU, but doesn't make any guarantees. Studio Lab is bound by capacity and demand. We are adding capacity all the time and we are working hard to make sure the distribution is as fair as possible. But there are times when the demand is greater than our capacity.

Last December, we did launch Notebook Jobs to help customers take their SageMaker Studio Lab Notebooks and run them easily in their AWS account. In this case, you are not relying on the SMSL capacity, but instead selecting (and paying) for any instance type you want. See this blog here: https://aws.amazon.com/blogs/machine-learning/run-notebooks-as-batch-jobs-in-amazon-sagemaker-studio-lab/ for more information. This can be powerful, and an inexpensive way to get the compute you need when you need it.

@JannicCutura, we are not having any problems with CPU. Can you confirm what problem you are getting? Thanks for the feedback. We are always striving to make this a better service for you. Michele SageMaker Studio Lab Product Manager

JannicCutura commented 1 year ago

Hey Michelle,

thanks for your quick reply!

I will look into your suggestions. In the meantime find attached my CPU problem.

Best,

Jannic

From: MicheleMonclova @.> Sent: Wednesday, 15 February 2023 19:56 To: aws/studio-lab-examples @.> Cc: Jannic Cutura @.>; Mention @.> Subject: Re: [aws/studio-lab-examples] There is no runtime available right now. Please change the compute type or try again later. (Issue #110)

Hi All, we are experiencing elevated error starting runtimes for GPU. The SageMaker Studio Lab team is working to restore the service.Please use CPU runtime until we are fully recovered. We apologize for any inconvenience.

The FAQ does state the upper limits for GPU, but doesn't make any guarantees. Studio Lab is bound by capacity and demand. We are adding capacity all the time and we are working hard to make sure the distribution is as fair as possible. But there are times when the demand is greater than our capacity.

Last December, we did launch Notebook Jobs to help customers take their SageMaker Studio Lab Notebooks and run them easily in their AWS account. In this case, you are not relying on the SMSL capacity, but instead selecting (and paying) for any instance type you want. See this blog here: https://aws.amazon.com/blogs/machine-learning/run-notebooks-as-batch-jobs-in-amazon-sagemaker-studio-lab/ for more information. This can be powerful, and an inexpensive way to get the compute you need when you need it.

@JannicCutura https://github.com/JannicCutura , we are not having any problems with CPU. Can you confirm what problem you are getting? Thanks for the feedback. We are always striving to make this a better service for you. Michele SageMaker Studio Lab Product Manager

— Reply to this email directly, view it on GitHub https://github.com/aws/studio-lab-examples/issues/110#issuecomment-1431856541 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLPQWUBEVDJ3HLNMYQWJATWXUREFANCNFSM5WGS575A . You are receiving this because you were mentioned. https://github.com/notifications/beacon/ADLPQWUSL7SQ3WAAGMZU5XTWXUREFA5CNFSM5WGS575KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOKVMGLHI.gif Message ID: @. @.> >

RaistlinD2x commented 1 year ago

Yeah this isn’t a GPU specific issue. The CPU instances are regularly unavailable. These issues are further impacted by TensorFlow bugs in 2.10. Colab uses 2.9.2 for the most stable version. These issues largely emerge when using transfer learning as there are conflicts with certain layers.

Again, it’s not GPU specific. I find the service unusable.

Jesse Richey


From: Jannic Cutura @.> Sent: Wednesday, February 15, 2023 1:56:42 PM To: aws/studio-lab-examples @.> Cc: Jesse Richey @.>; Comment @.> Subject: Re: [aws/studio-lab-examples] There is no runtime available right now. Please change the compute type or try again later. (Issue #110)

Hey Michelle,

thanks for your quick reply!

I will look into your suggestions. In the meantime find attached my CPU problem.

Best,

Jannic

From: MicheleMonclova @.> Sent: Wednesday, 15 February 2023 19:56 To: aws/studio-lab-examples @.> Cc: Jannic Cutura @.>; Mention @.> Subject: Re: [aws/studio-lab-examples] There is no runtime available right now. Please change the compute type or try again later. (Issue #110)

Hi All, we are experiencing elevated error starting runtimes for GPU. The SageMaker Studio Lab team is working to restore the service.Please use CPU runtime until we are fully recovered. We apologize for any inconvenience.

The FAQ does state the upper limits for GPU, but doesn't make any guarantees. Studio Lab is bound by capacity and demand. We are adding capacity all the time and we are working hard to make sure the distribution is as fair as possible. But there are times when the demand is greater than our capacity.

Last December, we did launch Notebook Jobs to help customers take their SageMaker Studio Lab Notebooks and run them easily in their AWS account. In this case, you are not relying on the SMSL capacity, but instead selecting (and paying) for any instance type you want. See this blog here: https://aws.amazon.com/blogs/machine-learning/run-notebooks-as-batch-jobs-in-amazon-sagemaker-studio-lab/ for more information. This can be powerful, and an inexpensive way to get the compute you need when you need it.

@JannicCutura https://github.com/JannicCutura , we are not having any problems with CPU. Can you confirm what problem you are getting? Thanks for the feedback. We are always striving to make this a better service for you. Michele SageMaker Studio Lab Product Manager

— Reply to this email directly, view it on GitHub https://github.com/aws/studio-lab-examples/issues/110#issuecomment-1431856541 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLPQWUBEVDJ3HLNMYQWJATWXUREFANCNFSM5WGS575A . You are receiving this because you were mentioned. https://github.com/notifications/beacon/ADLPQWUSL7SQ3WAAGMZU5XTWXUREFA5CNFSM5WGS575KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOKVMGLHI.gif Message ID: @. @.> >

— Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faws%2Fstudio-lab-examples%2Fissues%2F110%23issuecomment-1431943072&data=05%7C01%7C%7C7cd5e4bc8c064fe928f308db0f8ec2c9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638120878046482521%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dg8mzyY34RckuJx4K6RKT3Y8X%2F%2FWchSqXaIv61g82ds%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARSKXX2EAM2AMQMAUUOI6IDWXUYHVANCNFSM5WGS575A&data=05%7C01%7C%7C7cd5e4bc8c064fe928f308db0f8ec2c9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638120878046482521%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QCp%2FZhp5caiZcWuf6CL55sRY1BnVMoucMo1TVuJLOaY%3D&reserved=0. You are receiving this because you commented.Message ID: @.***>

rudro12356 commented 1 year ago

Hello,

I am trying to run a yolov5 custom object detection model. Is it possible to get GPU usage? It's for my school project. email: rudro12356@ku.edu

Thanks!

RaistlinD2x commented 1 year ago

If you can’t get access it’s not very expensive to stand up an instance in Studio that has a GPU so long as you shut it down when you’re not using it. Studio Labs though was unusable the whole time I was prepping for the TF cert.

Check this out to understand the free tier stuff. https://aws.amazon.com/pm/sagemaker/?trk=81375c4b-92b7-4225-90aa-3e1419735f36&sc_channel=ps&s_kwcid=AL!4422!3!544628448952!p!!g!!amazon%20sagemaker%20cost&ef_id=EAIaIQobChMIipmX9rTL_QIVRG1vBB3eRQIVEAAYASAAEgLXq_D_BwE:G:s&s_kwcid=AL!4422!3!544628448952!p!!g!!amazon%20sagemaker%20cost Jesse Richey


From: RokunuzJahan Rudro @.> Sent: Tuesday, March 7, 2023 7:31:21 PM To: aws/studio-lab-examples @.> Cc: Jesse Richey @.>; Comment @.> Subject: Re: [aws/studio-lab-examples] There is no runtime available right now. Please change the compute type or try again later. (Issue #110)

Hello,

I am trying to run a yolov5 custom object detection model. Is it possible to get GPU usage? It's for my school project. email: @.**@.>

Thanks!

— Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faws%2Fstudio-lab-examples%2Fissues%2F110%23issuecomment-1459129065&data=05%7C01%7C%7C7d36a88f5621464cfcb808db1f74d2e5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638138358827650067%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ieqwomJpQ3js5MwhsImnLv6kQ9GxhHxUiqYjz6UwIuM%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARSKXXY6KIVE3S3QOLIWHUDW27OOTANCNFSM5WGS575A&data=05%7C01%7C%7C7d36a88f5621464cfcb808db1f74d2e5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638138358827650067%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=uvcuRMpdv6Y2eg8FHSDC%2FM0AhmPV%2Bxs1tSEV1I17HC4%3D&reserved=0. You are receiving this because you commented.Message ID: @.***>

MicheleMonclova commented 1 year ago

Hey all. Today we rolled out CAPTCHA to battle the bots and scripts that absorb compute. Hopefully you have better access getting user sessions. Thank you for your patience.

HyunggyuJang commented 11 months ago

Still having problem connecting to GPU instance. So far, I have no luck to use GPU resource from Studio Lab :{

RaistlinD2x commented 11 months ago

Just give up on it, if you need a GPU instance either use your home PC using this: https://learn.microsoft.com/en-us/windows/wsl/tutorials/gpu-compute, or rent an instance on AWS.

I have the RTX3090 with 24GB of vRAM, I can training up to the GPT3.5XL I believe, somewhere around 1-1.5GB model size would be the maximum.

Alternatively you can try Colab with Google, I had more luck getting a GPU with them.


From: HyunggyuJang @.> Sent: Tuesday, September 19, 2023 10:10:08 PM To: aws/studio-lab-examples @.> Cc: Jesse Richey @.>; Comment @.> Subject: Re: [aws/studio-lab-examples] There is no runtime available right now. Please change the compute type or try again later. (Issue #110)

Still having problem connecting to GPU instance. So far, I have no luck to use GPU resource from Studio Lab :{

— Reply to this email directly, view it on GitHubhttps://github.com/aws/studio-lab-examples/issues/110#issuecomment-1726817058, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARSKXXYLFLVRGIIOSTB3JL3X3JNBBANCNFSM5WGS575A. You are receiving this because you commented.Message ID: @.***>

Vendetta-S commented 1 week ago

Sorry for necroing but that last message age like milk