search

aiidateam / aiida-tutorials

AiiDA tutorials web site
http://aiida-tutorials.readthedocs.org
23 stars 37 forks source link

💻 TECH: setting up machines in several regions to improve connection #273

Open ltalirz opened 4 years ago

ltalirz commented 4 years ago

We had one participant from China, whose download speed from the VM was effectively limited at 10 KByte per second [1]. While this is still enough to do to terminal commands over SSH, anything related to jupyter starts to be incredibly slow & it becomes impractical to, e.g. , download the export file with all the data at the end of the tutorial.

She also mentioned that she was not able to download Quantum Mobile directly but had some "service provider" take care of data mirroring before she could download it. And, of course, youtube is not available unless you know your tricks.

While we don't have the resources to cater individually to every geographical region, it might be worthwhile to think about what we can do to mitigate some of these issues for participants from China in the future.

@unkcpz Do you have any suggestions? E.g. is it possible to set up an automatic (or semi-automatic) mirroring of quantum mobile releases?

P.S. I just checked and this year we had just 2 participants from mainland China, but I suspect this will increase with time; in particular if we are able to improve the experience for them.

[1] I've experienced things like this as well when I was in China - for some reason, routes to certain locations use very large numbers of hops; plus there might be an imposed bandwidth quota.

unkcpz commented 4 years ago

hi @ltalirz glad to hear there are participants from China. I think three issues you mentioned have different solutions:

is it possible to set up an automatic (or semi-automatic) mirroring of quantum mobile releases?

ustc mirror supports new mirror request and it is one of the most stable mirrors I have used in China. The request guide is locate here at https://github.com/ustclug/mirrorrequest/ . I can help to make the request there but I think it is better from your official side, they may have a higher priority to considered the request from official source.

youtube is not available unless you know your tricks.

Sure, and I will keep on mirror the videos like here https://www.bilibili.com/video/BV1j4411f7jZ . After the virtual tutorial is over, could you compress the videos and upload somewhere so I can download them one time for all (previously I download from youtube one video a time with third-party web plugin which is really tedious:cry:)?

download speed from the VM was effectively limited at 10 KByte per second

You mean this step[1]? Where is the server held? I only know proxy as the way to overcome this problem, if you held the server in Europe which is not able to be reached quickly from Mainland China.

[1] https://aiida-tutorials.readthedocs.io/en/latest/pages/2020_Intro_Week/sections/retrieve_data.html#download-exported-archive

ltalirz commented 4 years ago

thanks a lot @unkcpz for the suggestions!

ustc mirror supports new mirror request and it is one of the most stable mirrors I have used in China. The request guide is locate here at https://github.com/ustclug/mirrorrequest/ . I can help to make the request there but I think it is better from your official side, they may have a higher priority to considered the request from official source.

ustc looks interesting - one question I have, though, is whether it is possible to automate this? I.e. would we have to request every time we make a new QM release, or can it be set up once and for all?

Sure, and I will keep on mirror the videos like here https://www.bilibili.com/video/BV1j4411f7jZ . After the virtual tutorial is over, could you compress the videos and upload somewhere so I can download them one time for all

Definitely, I will send you the link

You mean this step[1]? Where is the server held? I only know proxy as the way to overcome this problem, if you held the server in Europe which is not able to be reached quickly from Mainland China.

They were running in the Ireland region indeed. We could switch to a different region (such as US-EAST) next time if these can be reached better, but in that case we should make sure to optimise the experience for everyone.

We could e.g. think of considering 2-3 options and asking participants in the registration form to run a command like ping/wget to see how fast the different options would be for them.

unkcpz commented 4 years ago

ustc looks interesting - one question I have, though, is whether it is possible to automate this?

It should be, cause it also mirrors other large software base such as repositories of some Linux dist. But I guess may be only support automatically fetching files in same sub-directory? For example like this:

../
quantum_mobile/                                         00-Jul-2009 20:07                   - (and different release inside the directory)
conda/                                 00-Sep-2010 16:28                   -
core/                                 16-Jul-2009 05:42                   -

You can try to ask them about it, if you need any language help, let me know.

We could e.g. think of giving 2-3 options and asking participants to run a command like ping/wget to see how fast the different options would be for them.

Agree! From my experience servers locate in Japan and US is always much reachable from Mainland China.

ltalirz commented 4 years ago

@unkcpz I've opened the request https://github.com/ustclug/mirrorrequest/issues/273

Now we need some votes ;-) Perhaps mention it to Guoyu; we might also want to mention it to Jun Cheng's group from Xiamen University.

P.S. Their issue template currently contains the disclaimer

Notice: Our mirror service is currently short of available storage, so requests for new mirrors will be put on hold for a period. This shortage will be eased in the near future.

I.e. it might take a while. In any case, if you find an alternative, you can probably refer to this post since it should contain all the information necessary.

mbercx commented 3 years ago

I only quickly parsed the issue, but maybe this is something that should be opened on https://github.com/marvel-nccr/quantum-mobile? Pinging @chrisjsewell to see if he knows of any problems with downloading the QM.

However, something we should consider for the event is using several regions on AWS. People connecting from China may not have a very good speed to Ireland. @csadorf what do you think? Would this be a lot more expensive?

csadorf commented 3 years ago

However, something we should consider for the event is using several regions on AWS. People connecting from China may not have a very good speed to Ireland. @csadorf what do you think? Would this be a lot more expensive?

It probably would, but I can try to make an estimate. What I don't quite understand why the connection would be that bad. I would assume it to be much slower, but even when we deployed in Brazil I was able to get a decently fast connection here.

mbercx commented 3 years ago

True. I just remembered that also in last year's tutorial (where we used individual AWS instances) some participants complained about slow-ish connections. Also, would be cool to present this as another advantage of the kubernetes setup.

csadorf commented 3 years ago

I disagree with the prioritization here and have adjusted it. Further more, I am going to evaluate this, but realistically we are most likely not going to deploy in multiple regions.

mbercx commented 3 years ago

Fair! I do remember the connection being quite alright for Sao Paolo indeed, so I suppose it will not be a major issue. 👍

ltalirz commented 3 years ago

Just two small things to mention:

From my side this issue can be closed once we're making sure that the QM mirror will be up to date.