search

berkeley-dsep-infra / datahub

JupyterHubs for use by Berkeley enrolled students
https://docs.datahub.berkeley.edu
BSD 3-Clause "New" or "Revised" License
62 stars 37 forks source link

Enable QGIS for Civ Eng 70! #3575

Closed balajialg closed 1 year ago

balajialg commented 2 years ago

Civil Engineering 70 (https://classes.berkeley.edu/content/2022-fall-civeng-70-001-lec-001) requires students to launch QGIS application (from Datahub) to do their geospatial exploration as part of the Engineering Geology lab. Copying instructor @sitarn's words "In CE 70 I use an open source Python GIS code called QGIS, which I would like to set up for the class on DataHub. Having the code sitting on Data Hub - students able to access it and execute it from their own directory and then save the output would be fantastic". The instructor requires this setup during mid-semester but would love to test the setup by mid september!

  1. Is it possible to enable Jupyter-qgis set up in one of our hubs? In case I am not aware of QGIS being already enabled in our hubs, then please do let me know!
  2. Would this warrant creating a new hub (tailored towards Civil Engineering or a generic hub for launching desktop applications?) or 70+ students can use Datahub to launch QGIS? I am assuming the memory requirements in Datahub would not be conducive to launching this application. Please clarify if I am wrong.
ericvd-ucb commented 2 years ago

Looks like the Jupyer- Qgis implemenation from Qushing Wu is the way to go We need to figure out how to get a couple of plugins to install?

balajialg commented 2 years ago

Shared the mybinder link in this repository to launch QGIS with @sitarn. Got the following feedback about the latency and the overall QGIS functionality -

The desktop fires up instantly, without any latency. Also, all the functionality is there as far as I can tell. The only question is whose directory it is looking for files and then saving.

He is currently testing QGIS in binder extensively and will provide comprehensive feedback when he is ready.

yuvipanda commented 2 years ago

Yep, and I've already been extensively involved in it :D https://github.com/giswqs/jupyter-qgis/pull/1

yuvipanda commented 2 years ago

@balajialg my inclination is to make them a new hub!

balajialg commented 2 years ago

Sound good @yuvipanda. We should target creating this hub by mid-September so that the instructor has ample time to test the instructional workflow.

Couple of questions,

  1. Should we go for a generic URL like linuxdesktop.datahub.berkeley.edu to showcase our desktop environment or tailor this hub towards civil engineering (civeng.datahub) specific use cases? My inclination is to go for the former but open to your thoughts.
  2. What is the compute for the binder link that launches QGIS? That seems to work for the instructor.

Thanks to @ericvd-ucb for asking the maintainer to merge your pending PR yesterday.

yuvipanda commented 2 years ago

@balajialg the mybinder.org compute is significantly less than what we have for datahub.

My inclination is to have a civeng hub separately because it shows that we love them 😁

balajialg commented 2 years ago

@yuvipanda Oh wow, that sounds great! If that is the case it could be a Civil and Environmental Engineering focused hub (we did a workshop with them this January).

sitarn commented 2 years ago

Yes, that would be awesome … Nick

Sent from my iPhone

On Aug 17, 2022, at 8:32 PM, Balaji Alwar @.***> wrote:

 @yuvipanda Oh wow, that sounds great! If that is the case it could be a Civil and Environmental Engineering focused hub (we did a workshop with them this January).

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

yuvipanda commented 2 years ago

@balajialg awesome. what would it be called? civeng.datahub.berkeley.edu?

ryanlovett commented 2 years ago

Just some data points:

balajialg commented 2 years ago

@ryanlovett @yuvipanda Thanks for your input! Let's go with civeng.datahub considering the evangelizing aspect of creating a new hub dedicated to CEE and also the backing from SIS reference?

sitarn commented 2 years ago

Hi: I consulted with the Chair of the department as this is a bit of a "hot button issue," and if it is at all possible, please use CEE.datahub as that is properly reflecting the fact that the department has been Civil and Environmental for more than 20 years. Thanks, Nick

On Thu, Aug 18, 2022 at 9:32 PM Balaji Alwar @.***> wrote:

@ryanlovett https://github.com/ryanlovett @yuvipanda https://github.com/yuvipanda Thanks for your input! Let's go with civeng.datahub considering the evangelizing aspect of creating a new hub dedicated to CEE and also the backing from SIS reference?

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1220238421, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JBUILO2TP7DWZ7TC73VZ4E4TANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

fperez commented 2 years ago

BTW, I never ended up requesting qgis in stat159 b/c we had enough geo tools in Jupyter proper and that wasn't the focus of the course, but this is another example of the value of a fully-configured desktop with a good collection of tools.

Our JMTE hub does have it installed, and it works well! My gripes with it are:

But overall it's solid, and a perfect way to use the hub as a rich collection of tools to work with data that's in the cloud, and the hardware selection flexibility of the cloud.

I'm all for continuing to push forward a vision of our hubs as places to "live la vida nube" with all our computational needs, and not just "my notebook runner."

balajialg commented 2 years ago

Thanks, @sitarn for your feedback on the hub URL. We are targeting mid-September for launching the new hub.

@fperez Personally, super excited about the opportunity to use the Linux desktop environment in yet another course fulfilling your vision of "live la vida nube". What is the computational resource provisioned for the JMTE hub? Apparently, @sitarn highlighted that latency was not a huge challenge after exploring jupyter-qgis binder link. Curious about the resource provisioning for the new hub.

fperez commented 2 years ago

Oh, latency is just a bit of an issue if you're on a lousy internet connection. Since pure jupyter tools have their UI client-side, the connection quality is in general less of an issue. But VNC, while very efficient, is still more sensitive to connection issues, as all UI interactions and changes need to travel in real time back and forth.

That's all - it's just in the nature of VNC, nothing more (and not dependent on the backend resources).

But in general this works very well, and can be a life saver. In Stat 159, while we didn't use QGIS, having the Desktop option was fantastic.

balajialg commented 2 years ago

Ah, thank you so much @fperez for clarifying my misconception! This is something @sitarn should watch out for if the plan is to use QGIS real-time in a classroom setting (taking into account the recurring campus wifi issues).

sitarn commented 2 years ago

Thanks for the heads up. Actually, we will only use it for demonstration in class/labs and students will work on it on their own time for assignments and projects. So, we should be OK on bandwidth in the space where we plan to use it in the lab (we were a year ago). Also, a bit of latency is not an issue, comparing it with having to go from student to student to check their computer configuration and Python installation.

On Fri, Aug 26, 2022 at 1:53 PM Balaji Alwar @.***> wrote:

Ah, thank you so much @fperez https://github.com/fperez for clarifying my misconception! This is something @sitarn https://github.com/sitarn should watch out for if the plan is to use QGIS real-time in a classroom setting taking into account the recurring campus wifi issues.

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1228938200, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JBUC4O7W7XUUSEFDALV3EVEBANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

balajialg commented 1 year ago

@sitarn Apologies for the delayed response at our end. September was crazy. Thanks to @yuvipanda's work a new Civil Engineering hub got deployed today. You can access it via this URL. You can launch the QGIS application through the below steps,

  1. Login using your bcourses credentials. It takes you directly to the bcourses application.
  2. Launch QGIS image

Please test the application and let us know if it works well. You can reopen this issue if there are any further issues. Thanks!

sitarn commented 1 year ago

Outstanding! Thank you very much and it is just in time. Just one question: How do the students access the software? Using the same link procedure as I did? Thanks again for the help. Nick Sitar

On Thu, Oct 6, 2022 at 12:49 PM Balaji Alwar @.***> wrote:

@sitarn https://github.com/sitarn Apologies for the delayed response at our end. September was crazy. Thanks to @yuvipanda https://github.com/yuvipanda's work a new Civil Engineering hub got deployed today. You can access it via the URL https://cee.datahub.berkeley.edu/. You can also launch the QGIS application through the below steps,

  1. Login using your bcourses credentials.
  2. Select New -> Desktop option (New appears in the top right corner) [image: image] https://user-images.githubusercontent.com/2306166/194405217-b745d6f2-a3e8-472f-a2a9-6457612a893f.png
  3. Launch QGIS [image: image] https://user-images.githubusercontent.com/2306166/194405273-2a17be6e-ec65-442c-af5d-90923cf4683a.png

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1270601979, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JC7M5FMROVRL5L2PE3WB4UOJANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

balajialg commented 1 year ago

@sitarn Thanks for testing QGIS immediately! Yes, Students will also follow the same process as you followed. Login to the CivEng hub and launch the QGIS application. A couple of questions,

  1. Are you comfortable with the current version of QGIS?
  2. You were looking at QGIS plugins as part of your requirement. Are you able to play with those plugins? Any feedback?
sitarn commented 1 year ago

Thanks. I am working on it. Pls give a bit of time as I slot this in with my other fun entertainment...I suspect I may have questions but so far so good

On Fri, Oct 7, 2022 at 11:45 AM Balaji Alwar @.***> wrote:

@sitarn https://github.com/sitarn Yes, Students will also follow the same process as you followed. Login to the CivEng hub and launch the QGIS application. A couple of questions,

  1. Are you comfortable with the current version of QGIS?
  2. You were looking at QGIS plugins as part of your requirement. Are you able to play with those plugins? Any feedback?

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1271958958, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JAPYYSR4Q5ZYSN7N4TWCBVUFANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

sitarn commented 1 year ago

Hi: First round of feedback: 1) It would be great to have a more recent version of QGIS the Las Palmas is "ancient" in that the available plugins are somewhat limited for what we need 2) The way the site is configured, I cannot access files on my computer, nor do I see a way to create my own directory with the necessary files. That may be my lack of understanding of the configuration. Thanks, Nick

On Fri, Oct 7, 2022 at 11:45 AM Balaji Alwar @.***> wrote:

@sitarn https://github.com/sitarn Yes, Students will also follow the same process as you followed. Login to the CivEng hub and launch the QGIS application. A couple of questions,

  1. Are you comfortable with the current version of QGIS?
  2. You were looking at QGIS plugins as part of your requirement. Are you able to play with those plugins? Any feedback?

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1271958958, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JAPYYSR4Q5ZYSN7N4TWCBVUFANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

balajialg commented 1 year ago

@sitarn Thanks for the feedback.

  1. Yes, we will upgrade QGIS to the latest version asap.
  2. Can you use this documentation to understand how to upload files to a specific directory in the Jupyterhub account? It should be a straightforward process. Please let me know if you face any difficulties.
yuvipanda commented 1 year ago

@sitarn try now, we should be up to qgis 3.22

sitarn commented 1 year ago

Great, the new version of QGIS has the options we need. I was able to upload files, although it seemed a bit awkward: manually go to the /tree directory on the Jupyter notebook and then place all the files on the desktop, since that is the only directory I can access from QGIS. However, QGIS then worked as advertised. I was able to upload the necessary plugins and perform basic functions. In that context one more question. Is each instance of QGIS unique to the each user, ie each user has to do the basic configuration? Thanks a lot. Nick

On Mon, Oct 10, 2022 at 1:43 PM Balaji Alwar @.***> wrote:

@sitarn https://github.com/sitarn Thanks for the feedback.

  1. Yes, we will upgrade QGIS to the latest version asap.
  2. Can you use this documentation https://tljh.jupyter.org/en/latest/howto/content/add-data.html to understand how to upload files to a specific directory in the Jupyterhub account? It should be a straightforward process. Please let me know if you face any difficulties.

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1273802080, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JGYXJZ2VNTQUK2QDJTWCR5WZANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

sitarn commented 1 year ago

Yes, that version is perfect. Thank you very much!

On Mon, Oct 10, 2022 at 4:06 PM Yuvi Panda @.***> wrote:

@sitarn https://github.com/sitarn try now, we should be up to qgis 3.22

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1273895579, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JGAVHHSFORU7YME4LDWCSOQDANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

yuvipanda commented 1 year ago

@sitarn each instance is unique to the user, nothing is shared. Can you share what files you would want to distribute? We generally distribute files to users via GitHub and 'nbgitpuller links' - try clicking the 'homework' link in http://data8.org/fa22/ to see how that works. We can use a similar setup to distribute files for you too, depending on how big they are.

sitarn commented 1 year ago

Ok, thanks. Let me "play around" first, since getting the files from a public source is a part of the exercise. However, there will be a basic data set that I normally set up on bcourses, so setting up via GitHub makes sense. Thanks for all the help. Nick

On Mon, Oct 10, 2022 at 8:07 PM Yuvi Panda @.***> wrote:

@sitarn https://github.com/sitarn each instance is unique to the user, nothing is shared. Can you share what files you would want to distribute? We generally distribute files to users via GitHub and 'nbgitpuller links' - try clicking the 'homework' link in http://data8.org/fa22/ to see how that works. We can use a similar setup to distribute files for you too, depending on how big they are.

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1274028629, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JEXGTRMX2FTVRQCYGTWCTKWTANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

sitarn commented 1 year ago

Well, since there is a choice, we would actually prefer cee.datahub.berkeley.edu. The reason being that we are Civil and Environmental Engineering (CEE) but for historic reasons the other abbreviations stick. Thanks, Nick

On Thu, Aug 18, 2022 at 12:29 AM Ryan Lovett @.***> wrote:

Just some data points:

  • their website uses CE as the department in course names https://ce.berkeley.edu/programs/geo/courses, and the departmental website itself is ce.berkeley.edu, so maybe ce.datahub.berkeley.edu? This mirrors publichealth.berkeley.edu / publichealth.datahub.bekeley.edu, and dlab, and eecs.
  • SIS uses CIVENG https://classes.berkeley.edu/content/2022-fall-civeng-173-001-lec-001, so that's a another vote for civeng.datahub.berkeley.edu. civeng is clearer than ce for people outside of that unit. A counterpoint is that Public Health is PBHLTH in SIS but we use "publichealth".

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1219132662, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JB6PVBT7XOG7SPXOH3VZXQ6JANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

yuvipanda commented 1 year ago

@sitarn it is already at cee.datahub.berkeley.edu/

balajialg commented 1 year ago

@sitarn If you need help with sharing the files with your students as per @yuvipanda's suggestion, please do let me know. I can schedule some time to help you set up the GitHub repository (with the needed files) and distribute it using unique links.

sitarn commented 1 year ago

Perfect. Thank you very much. I will put the code through all the paces of what I expect the students to do and I expect I will be back with questions. However, the code works perfectly.

On Tue, Oct 11, 2022 at 11:59 AM Balaji Alwar @.***> wrote:

@sitarn https://github.com/sitarn If you need help with sharing the files with your students as per @yuvipanda https://github.com/yuvipanda's suggestion, please do let me know. I can schedule some time to help you set up the GitHub repository (with the needed files) and distribute it using unique links.

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1275141608, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JE5DV7D7VHYQMGABN3WCW2KTANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

balajialg commented 1 year ago

@sitarn How was your experience using this hub for your coursework? Would love your feedback. Meanwhile, can we also close this issue if there aren't any more updates?

sitarn commented 1 year ago

Hi: Thanks for checking. The full test starts next week. So, far all works perfectly for me, but then I am old school, i.e. the computer should do what I want rather than the other way around. Now we'll see how the students handle it. So, I have not bothered you until I see the student experience. On a separate note, I tried to use the Jupyterhub for straight Python scripts for my groundwater class and I find that I have to install (pip install scipy) every time I reopen the server connection. What am I missing? Thanks a lot, Nick

On Thu, Nov 3, 2022 at 9:10 AM Balaji Alwar @.***> wrote:

@sitarn https://github.com/sitarn How was your experience using this hub for your coursework? Would love your feedback. Meanwhile, can we also close this issue if there aren't any more updates?

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1302340609, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JDDYR3OF3FF5SVW2KTWGPPW5ANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

yuvipanda commented 1 year ago

@sitarn if you look at https://github.com/berkeley-dsep-infra/datahub/tree/staging/deployments/cee/image, you'll see we have basically installed no real scientific python libraries there :) If you want, you can make a PR adding those to the environment.yml file there, and that should add whatever you need

balajialg commented 1 year ago

@sitarn reported that the setup overall worked great this semester (Thanks a ton for setting this up @yuvipanda). However, there were latency issues with QGIS that students reported for a few days. I am not sure whether this issue is due to the slowness of hubs reported by other users like #3969 or whether the CEE hub requires increased compute.

In his words,

times were pretty much afternoons 3~8pm, Nov. 22-24, and the problem was the response time of QGIS. I thought it could have been the internet connection in Davis Hall, but students had slow response issues at other locations on campus. I experienced a bit of it, but not the extreme that I saw on some of the student desktops.

@yuvipanda Do you have any suggestions for us here?

yuvipanda commented 1 year ago

It would be useful to know what they mean by 'slow connection', as it could mean many things:

  1. qgis is not responding to clicks and what not in time. This could be due to not enough CPU for qgis, due to network latency, etc
  2. Servers not starting quick enough.
  3. The operations being performed in qgis itself not being fast enough - say it isn't loading a map fast enough. This could also be CPU.

So having more detail would help.

balajialg commented 1 year ago

Thanks, @yuvipanda. I am guessing that it should be scenario 1 or 3. @sitarn Would appreciate your input here.

sitarn commented 1 year ago

The problem was a very slow response by QGIS on data import requests from the folders on the desktop. It looked like the app completely froze for significant periods of time. This was across multiple laptops, PC's and Macs, mostly pretty capable. Once the layers were imported into QGIS, the response was normal and there were no additional issues. Nick

On Mon, Dec 12, 2022 at 2:03 PM Yuvi Panda @.***> wrote:

It would be useful to know what they mean by 'slow connection', as it could mean many things:

  1. qgis is not responding to clicks and what not in time. This could be due to not enough CPU for qgis, due to network latency, etc
  2. Servers not starting quick enough.
  3. The operations being performed in qgis itself not being fast enough
    • say it isn't loading a map fast enough. This could also be CPU.

So having more detail would help.

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1347402940, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JE6OXQJTRIUKNPGEELWM6OL7ANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

fperez commented 1 year ago

@sitarn - do you have a sense of how big the layers were, and in what format? Filesystem-intensive operations on the hub can be exceedingly slow in some cases, leading to what may appear as entirely stalled systems. To give you an idea, I've seen a (complex) conda environment build that takes ~90s on my laptop take ~30 minutes on a hub. That's probably an extreme case (env builds involve tens of thousands of files, so it's super filesystem-intensive), but it can give you a sense of the problem.

sitarn commented 1 year ago

The large folders/files are 16-20 MB. The interesting thing is that they often load up really fast. I just checked on my iMac in my office using eduroam wifi and the response was instant. So, that is why I am thinking that it was some aspect of overload in the system.

On Tue, Dec 13, 2022 at 11:08 AM Fernando Perez @.***> wrote:

@sitarn https://github.com/sitarn - do you have a sense of how big the layers were, and in what format? Filesystem-intensive operations on the hub can be exceedingly slow in some cases, leading to what may appear as entirely stalled systems. To give you an idea, I've seen a (complex) conda environment build that takes ~90s on my laptop take ~30 minutes on a hub. That's probably an extreme case (env builds involve tens of thousands of files, so it's super filesystem-intensive), but it can give you a sense of the problem.

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1349538870, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JHIT3EIPTQGSFD2DBDWNDCUDANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***

fperez commented 1 year ago

Yes, I would then hazard a guess that it's contention over the filesystem, from many users hitting it at the same time. NFS in the cloud isn't exactly nice, and many users hitting the NFS server simultaneously, even with fairly moderate sized requests, can also probably induce quite a bit of latency.

But now I'm guessing, so I'll leave it to the team to provide you properly informed responses :)

balajialg commented 1 year ago

Thanks, @fperez and @sitarn for your inputs! If NFS server calls is the main reason for this issue then I am making an assumption that it may not pose a similar challenge as we decided to move to Google File Store next semester. It could potentially solve for this problem or maybe we need to do more optimizations so that QGIS can work well. Let's observe how the the first few weeks of the Spring semester and take a call @sitarn @yuvipanda?

EECS instructors used pyqt5 in their Linux desktop environment and apparently had a seamless experience. I am curious to understand whether their use case involved any file system calls.

balajialg commented 1 year ago

@sitarn Hope you are having a good start to Spring semester! I am closing this issue as the intended objective of enabling QGIS in Civ Eng hub got completed. If there are any performance related issues then we can open a new issue and track it. Thanks!

sitarn commented 1 year ago

Thank you & all the best going forward. Nick

On Mon, Jan 23, 2023 at 2:32 PM Balaji Alwar @.***> wrote:

@sitarn https://github.com/sitarn Hope you are having a good start to Spring semester! I am closing this issue as the intended objective of enabling QGIS in Civ Eng hub got completed. If there are any performance related issues then we can open a new issue and track it. Thanks!

— Reply to this email directly, view it on GitHub https://github.com/berkeley-dsep-infra/datahub/issues/3575#issuecomment-1401099776, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXP7JBGSML2J3XMUVFUAQLWT4BH5ANCNFSM56TBJK3Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Prof. Nicholas Sitar, Ph.D. Edward G. Cahill and John R. Cahill Chair and Distinguished Professor of Civil and Environmental Engineering 449 Davis Hall UC Berkeley Berkeley, CA 94720 @.***