Vision Walk

This repository contains the complete source code for the Vision Walk system.

View Demo . Report Bug . Request Feature . Teaser Video Youtube Link . Teaser Video Drive Link

Table Of Contents

• About the Project
• Built With
• Getting Started
  • Prerequisites
  • Installation
• Usage
• Roadmap
• Contributing
• Authors

About The Project

Three components of Vision Walk are:

• iOS end user system that automatically captures the user environment and describes it.
• A python server that processes and compresses the received uncompressed JPEG image.
• An Inference server: BLIP captioning model that processed the image and returns the caption.

Built With

The development of the application was made possible by following tools and technologies.

• Swift
• Python
• Xcode
• Instrument
• BLIP Image Captioning Large
• Google MLKit Translate
• Postman
• ngrok
• Amazon Web Services

  Getting Started

To get a local copy up and running follow these steps.

Prerequisites

1. Setup a Virtual Environment (venv)

python3 -m venv <venv-name>

2. Activate the virtual environment

source <venv-name>/bin/activate

3. Install the required packages

pip install transformers torch pillow flask

4. Install Brew utility

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

5. Install cocoapods

brew install cocoapods

6. Install ngrok

brew install ngrok/ngrok/ngrok

7. Install Xcode, command line tools and the iOS SDK.

8. Generate Ngrok auth key from the ngrok website.

9. Enter your ngrok API key

ngrok config add-authtoken <your-auth-key>

Installation

1. Clone this repository.

https://github.com/shashank6341/Vision-Walk-HCI.git

2. Navigate to the repository

3. Install PODS in the iOS folder.

pods install

4. Run the project with Vision Walk.xcworkspace

git clone https://github.com/your_username_/Project-Name.git

3. Start the blip_caption_server.py server

python3 blip_caption_server.py

4. Expose your local python server through ngrok.

ngrok http <port-no>

Usage

The system is in experimental stage and can infer incorrect results. Please test the system in safe environments.

Launch the system and it will automatically capture the surrounding environments. Test across various scenarios and raise a new issues incase you encounter any issues or have improvements suggestion.

Add system demo images For more examples, please refer to the Documentation

Roadmap

See the [open issues](https://github.com/shashank6341/Vision Walk HCI/issues) for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

• If you have suggestions for adding or removing projects, feel free to [open an issue](https://github.com/shashank6341/Vision Walk HCI/issues/new) to discuss it, or directly create a pull request after you edit the README.md file with necessary changes.
• Please make sure you check your spelling and grammar.
• Create individual PR for each suggestion.

Branch Pattern

Create a new branch for each new feature or a sub-feature for easy debugging.

• feature/{feature-name}
• Once tested and working as expected.
• Create a PR from feature to main.

Creating A Pull Request

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Authors

• Shashank Verma - Concordia University
• Ayushi Chaudhary - Concordia University
• Het Dalal - Concordia University
• Khyati Bareja - Concordia University
• Kenish Halani - Concordia University
• Riddhi Bhuva - Concordia University
• Rohit Rohit - Concordia University