search

exo-explore / exo

Run your own AI cluster at home with everyday devices 📱💻 🖥️⌚
GNU General Public License v3.0
10.76k stars 613 forks source link

Questions and Future Requests and Feasibility #116

Open kuri54 opened 2 months ago

kuri54 commented 2 months ago

@AlexCheema I am developing AI for medical applications. In order to use AI, especially LLMs and VLMs (including MMLLMs), in clinical settings, the following are extremely important:

By using this library, the likelihood of not having to prepare large servers within the hospital and not placing data containing personal information on cloud servers has greatly increased. Therefore, I am considering the following setup, but is it possible in the future? Additionally, do you have any plans for this?

  1. Place multiple Mac minis or Mac Studios on the same network (assuming an in-house network) and connect them with Thunderbolt cables.
  2. Run the main.py of exo on the devices from step 1 and build a ring memory.
  3. Connect an iPhone (on the same network) in the examination room to the ring memory built in step 2.
  4. Attach the iPhone to a microscope and infer the images seen through the microscope using, for example, LLaVA. The inference can be done on individual images, but ideally, it would infer the images displayed on the iPhone in real-time.

If this setup is feasible, it would be a major breakthrough in the medical field. It means that it would be economical, free of the stress of handling data externally, and user-friendly.

Currently, code to build ring memory on mobile devices like iPhones is not provided, but I believe it will be available in due time. Is the above scenario feasible? If you have any plans or ideas, it would be extremely helpful to know. We have high expectations for this library.

clover1980 commented 2 months ago

Macs are not server-grade equipment, Apple discontinued server-based OS X many years ago, so i'm not sure what manager will approve such expenses on consumer devices for enterprise entity, that's the core problem, they must migrate to Linux fast! It's like using consumer-grade Jeep instead of tractor on farmfield. I have Mac mini myself and it's overheating like oven (after thermal paste change & without bottom lid, worst device i've used).

kuri54 commented 2 months ago

Thank you for your input. While your concerns about the hardware are valid and worth considering, they are more relevant as feedback for Apple regarding their products. I recommend sharing your experiences and suggestions directly with Apple through their review channels.

However, my question and request are specifically about the development and potential future capabilities of this library. I am focused on its application in a clinical setting where fast, real-time detection, local operation, and cost-effectiveness are crucial. If you have insights or suggestions related to these aspects or the library’s capabilities, I would greatly appreciate your constructive feedback. I am particularly interested in hearing constructive opinions from the development team regarding the feasibility and future plans for the library.

AlexCheema commented 2 months ago

Hey @kuri54 this sounds like a good use-case of exo. exo allows you to set up an on-premise AI cluster easily without needing a dedicated person or team to set up and maintain it.

I'll post a demo later today showing something similar where an iPhone can be used to trigger a VLM and LLM on your local cluster.

ProjectAtlantis-dev commented 2 months ago

we haven't had any heating issues w the mac studios - certainly not compared to the nvidia cards which are just a fire hazard imo

kuri54 commented 2 months ago

@AlexCheema Thank you so much for your response! I’m really looking forward to the demo. I’m excited about the possibility of making this use case a reality. I appreciate your support and look forward to seeing how it unfolds.

kuri54 commented 2 months ago

@ProjectAtlantis-dev Thank you for sharing your experience! In our environment, the Mac Studio has also been running very quietly and without generating significant heat, which is great. We’re aware of the considerable heat output from high-end NVIDIA cards as well, and it’s definitely something we have to manage carefully, especially when it comes to room temperature, which is a concern for us too.

wanghaisheng commented 3 weeks ago

Hey @kuri54 this sounds like a good use-case of exo. exo allows you to set up an on-premise AI cluster easily without needing a dedicated person or team to set up and maintain it.

I'll post a demo later today showing something similar where an iPhone can be used to trigger a VLM and LLM on your local cluster.

@AlexCheema any progress