search

daniel-birket / audible-adventure

An accessible Adventure-like game & engine with a hybrid text & audio user interface.
GNU General Public License v3.0
3 stars 0 forks source link

Fledgling audio game project for blind #2

Closed daniel-birket closed 1 year ago

daniel-birket commented 1 year ago

Hello, contributors to 'lucia',

I am Daniel Birket, an old sighted software engineer. I'm writing this issue with your usernames at-mentioned to introduce you to my friend @lawrenceper , a blind college student who aspires to learn programming and write accessible software, particularly for music. He is a composer and digital musician. (Listen to 'DJ Allegretto' on YouTube.) We met some years ago through the Be-My-Eyes app when he needed a human screenreader to help install a Linux distro. Since then, I've served as a remote sighted aide to his studies.

My hope is to connect Lawrence with a community of programmers here on GitHub that share his interests and that can serve as role-models and help him to follow his dreams, despite the difficulties presented by the frustratingly ignorant and apathetic sighted world. Please say hello.

Lawrence has recently been studying Python online. In the interest of applying what he is learning and introducing him to more rigorous programming techniques than text-book examples, I have proposed that we collaborate to create a moderately-simple text and audio game in the vein of the classic "Colossal Cave Adventure" text adventure. (Please see the requirements document in this repository's design folder for details.) I invite you to follow this repository and offer encouragement, advice and help.

I have noticed, among your collective repositories, that there are several audio games, engines and libraries in addition to 'lucia'. I plan to review them for possible inclusion or just to study for techniques. As yet, we are just starting on this project. I also have much to learn as most of my 40 years of software experience is in a unique field. I have only written a quick weekend video game in python just to get the feel of it. Your advice and feedback is welcomed.

@amerikranian, thank you for writing https://amerikranian.github.io/Python-guide/ . It looks very helpful.

Daniel Birket

daniel-birket commented 1 year ago

Adding more at-mentions to cast a wider net for people with interest in programming by and for the blind:

Please see the issue description above and this repository, particularly the requirements doc in the design folder. Please say Hello to @lawrenceper, a student programmer who wants to write accessible software.

TheQuinbox commented 1 year ago

Hi, I'd also be willing to help out with this. Looks like you've got a pretty decent start so far. I'm most familiar with Python, TypeScript, and C#/F# with the DotNet framework. I've worked along side many extremely skilled blind hackers throughout the past couple of years, and thing we can possibly do very big things with this project :)

daniel-birket commented 1 year ago

@TheQuinbox , Thank you for the offer of help and words of encouragement. I'm still feeling my way through some early design choices, including how to produce the audio. (I'm leaning toward SDL2 so far, as I think multi-channel sound will be needed. playaudio and simpleaudio are single channel and pygame mixer just seems to wrap SDL.) I've got a lot of user interface details to discuss with Lawrence and an entire app to decompose into Classes and functions. As we work through the design, I'll invite your feedback. Thanks.

lawrenceper commented 1 year ago

Hi all,

Thank you so much for helping me learn computer programming via this project. I am super excited to get started learning and putting things together.

A bit of background information about me: my name is Lawrence and I am a musician, I play the piano, sing, and produce my own music with computer software like audacity and QuickWindows sequencer. software like audacity and quick window sequencer help me as a blind person make music, something that I thought was never possible until I kept practicing and tinkering around with audio and musical instruments. When I made my first song, and finished putting it together by myself, I realized that music production via a computer, with the right software, it's certainly possible. My ultimate goal is to expand the capabilities of modern smartphones, by creating an app that is accessible for blind people, that will make it much easier to compose, and even produce, music on a mobile device.

I am also very interested in computers and all types of technology. I have been interested in the field of computer programming since 2013. I am learning python via a DIY programming class, but I haven't learned The development and design aspects of software development, something that I'd like to learn via this project. Since I am still a beginner, my focus right now is on python, but I am exploring other languages like JavaScript, HTML, CSS, and C++. In addition, with my new raspberry pi 400, I hope to learn more about Linux and working with a command line, including working with Git. :-)

I understand that DIY classes are more challenging than attending a college or university, but I am very motivated to continue learning and expanding my knowledge in Computer programming and software development.

looking forward to working with all of you on this project.

TheQuinbox commented 1 year ago

@daniel-birket, I'd actually recommend Bass for an audio library. Supports all kinds of formats, and has plenty of effects and such. I personally use sound_lib as a Bass wrapper (http://www.github.com/accessibleapps/sound_lib).

Amerikranian commented 1 year ago

@daniel-birket and @lawrenceper,

First, I am happy that my guide seemed to be semi-helpful. To be honest, I intended to add more sections concerning game design, at least from my personal experience, but the community reception of the guide--indifference--slightly decreased my motivation for doing so.

Second, allow me to be frank. I am currently starting my Freshman year at college in the Computer Science major and am frankly unsure how much free time, if any, I am going to have. I would like to help in from I can, but I will not be able to estimate the capacity of my assistance until we are 2 or so weeks into the semester (roughly three from the writing of this post).

Third, allow me to provide some suggestions and overall remarks regarding your design: You mentioned requiring audio as one of your communication interfaces and have brought up SDL's channel mixer. However, depending on how advanced you may want it to be this may not be enough. @TheQuinbox gave you an option of the Bass library, and I can concur that it would work reasonably well. However, if you want things like HRTF, it may not be enough. I suggest considering what exactly would be communicated with audio and / or how detailed would you want it to be (Do you want the player to feel like they are standing within an environment, or are you okay with just specifying the pan / volume a sound plays at. How advanced do you want your effects to be? Would you like filtering? Reverb?) and so on. I would also consider defining exactly how your game interface will look like. You mentioned speech recognition, but last time I looked at it I found that you either would invest in an API or have limited requests for all the users of the app. I also do not recall seeing asynchronous libraries for handling the parsing of audio data, which could add extra complexity due to the need to employ threading. The real question is how set are you on including a speech backend as one of the game's methods of accepting input? I would be curious as to your thoughts on the matter. Unfortunately, most library GUI options and or shapes are not accessible out of the box. For instance, if you drew a rectangle with SDL with the words "Start Game" inside it or created a button with Pygame, our screen readers won't be able to access anything because said drawings are on Pygame's / SDL canvas and are not actual native objects. There is also the problem of differing screen reader implementations where one will pick up what the other misses and vice versa. This is especially noticeable on Linux where visually impaired users would have trouble accessing an app written an Electron vs. them being perfectly capable of using it on Windows. I would be interested regarding your thoughts on a slightly modified text terminal--we can still offer autocompletion and keyboard shortcuts but it will require employing different tools than the standard input function. Addition of graphics may be complicated, however. It is totally possible to write an application that is cross-platform and accessible at the same time while using things like Pygame / SDL displays, it's just tricky because you would need to figure out how to define parent / child hierarchy and then provide an ellegant way of navigating it with both the keyboard and the mouse. You would also need to hook in something that communicates to screen readers, though that is actually the easy part, see https://github.com/accessibleapps/accessible_output2. Using this library will ensure compatability with speech backends on Windows, Mac, and Linux, but as of now there is no support for IOS, Android, or Windows Smart Phones (I believe I saw this in your design doc).

As you can see, this all comes down to the effort you wish to put into it.

I am sure others will chime in here, those with likely more experience than me, but this should get the conversation started.

ultrasound1372 commented 1 year ago

Alas I won't be of much use here. My inclusion on the Lucia project was because it was born out of a discussion in a Skype group some years ago I was a part of, but it suffers from something that plagues the audiogame community. Nearly everyone is a hobbyist, teams are uncommon, and many among us have no formal education in the design of such things before we start building frameworks, which leads to mountains of spaghetti code adhering to no style guide in particular. I am also not competent enough with Python, my inclusion on the Lucia project was mainly for writing documentation. I am a hobbyist musician and a full time student in the Fall and Spring semesters, as well as being one of three developers and maintainers of a multiplayer game at the moment. I concur with the choice of Bass, though, from experience. I and others have been wondering about FMOD, but many mainstream frameworks just have not made it into our community yet. Many of us, including myself, are primarily fluent in something called BGT, an abandoned scripting language built with AngelScript which comes with a basic standard library that can be used to create audiogames, and not much else. All efforts for building audiogames in other languages have either not gotten off the ground or are within select teams, Lucia being an example of the former last I was aware. Existing game engines are exceedingly difficult to work with in the ways we're used to, the most common being UI as was said before. Within audiogames the semantics of user interfaces must be implemented from scratch and carry simulated behaviors to the speech output and user interaction, which is just frustrating to work with. The only truly accessible UI framework in Python I'm aware of is WXPython, followed distantly by recent versions of QT. And good luck making a game with one of those, they just weren't built for it. If you wish to tackle accessibility from the sighted programmer's end, check out the developer of the game Sequence Storm, available on Steam and itch.io. But even with that, the only speech output is provided by the system's native speech APIs, not the screen reader in use. In many cases this actually requires the user to disable their screen reader to play the game. Accessible Output 2 is a good solution for this as they figured out how to get VoiceOver output, though it requires the enable VoiceOver to be controlled by AppleScript setting to be enabled. And Linux is all just guesswork, as it is for many other things.
This message has gotten a bit rant-y given I'm providing context for you as someone who is not intimately familiar with our conditions, so I apologize. I would be willing to test this game and give intuitive feedback on how some interactions should probably work, but I don't believe I would be of any use for introducing someone to making an audiogame. I don't think that highly of my sound design skills either, I could maybe make you music though!

daniel-birket commented 1 year ago

@Amerikranian and @ultrasound1372,

Thank you for chiming in. The apparent existence of a disjointed audio-game community, as well as a the hints of the challenges of a blind programmer are very helpful, particularly from my sighted perspective.

I note that both of you are college students, like Lawrence, so I urge you to give those studies priority. Games and programming project should take a back-seat, unless they also serve to advance your education.

@Amerikranian, your notes on designing for this genre give me more reasons to keep it simple. Remember that my ulterior motive here is to give Lawrence (and anyone else of similar limited experience and ability) an opportunity to gain experience. As his programs so far are near the hello-world end of the spectrum, We should keep it simple, except perhaps on the audio bits, where his skills exceed mine. Perhaps there will be another design iteration someday that builds on what we learn with this simple experiment. Personally, I'd love to be part of a team that creates an eventually-popular framework or engine for audio-games, but I still have a day-job.

Considering the feedback from all of you, this is where I'd like to take this interface and design:

ultrasound1372 commented 1 year ago

If using BASS or something that supports DirectX you would be able to add reverb, although I'm unsure of how this works on other platforms. It's sort of based on EAX which nobody implements anymore, but there are software implementations which at least BASS uses, unsure about Core Audio and the like equivalents to DirectX. I do believe BASS would be easier to work with in this case than FMOD. If you look at some of Lucia's code you'll find a thing called the sound pool, which is among the standard primitives blind programmers expect. It handles the translation of cartesian coordinates to pan/volume deltas to give the illusion of motion, though indicating something is behind you in true 3D, or below you in 2D platforming, is difficult and is usually approximated by a slightly decreased pitch. Lucia's include is based off code that I believe @Amerikranian and @cartertemm ported to python from one of many individual modifications to the standard BGT sound pool that add 3d and rotational support.

lawrenceper commented 1 year ago

I don't know if direct X is compatible with other platforms like macOS and Linux (E.G., Debian, Ubuntu, fedora, etc.). I, myself, use either a MacBook Pro or a raspberry pie running the default raspberry pie OS based on Debian bull's-eye. I have experimented with the sox command-line tool, would it perhaps be possible to send audio via libsox and python, and add reverb there? I know of a Linux-based screen reader called Fenrir that uses libsox to play sounds.

Thank you.

TheQuinbox commented 1 year ago

DirectX isn't a necessary component (only on Windows is it used). Bass is entirely cross-platform, and I've seen it work perfectly on Windows CE, macOS, and Ubuntu.

ultrasound1372 commented 1 year ago

Sox alone wouldn't be able to do it no. You'd likely want to use BASS and pair that with ALSA or the like.

akash07k commented 1 year ago

Good idea, will clone the project and will try to contribute

daniel-birket commented 1 year ago

Thank you. It may be early to fork it. I’m still finding my footing. Please follow.

From: Akash Kakkar @.> Sent: Saturday, August 13, 2022 3:05:27 AM To: daniel-birket/audible-adventure @.> Cc: Daniel Birket @.>; Mention @.> Subject: Re: [daniel-birket/audible-adventure] Fledgling audio game project for blind (Issue #2)

Good idea, will clone the project and will try to contribute

— Reply to this email directly, view it on GitHubhttps://github.com/daniel-birket/audible-adventure/issues/2#issuecomment-1213878377, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AALXDZCE6LTKR7PR6H3XGLTVY5CLPANCNFSM56JBFRKA. You are receiving this because you were mentioned.Message ID: @.***>

lawrenceper commented 1 year ago

Hello all, I would like to thank those who contributed suggestions for Python audio libraries for this project. We have made a decision to use the SDL2 library based on the following criteria: Open Source: the library should be open source or public domain so it doesn’t conflict with the GPL license. Although the bass library can be used for nonprofit uses, the library is commercial, and thus, is not open source. Cross-platform: the library should be cross-platform, meaning that it can be used on multiple operating systems. The SDL library works with Windows, MacOS, and Linux. Works in the command-line: Some audio libraries that we have researched have the ability to make audible menus, ETC. We envision the game running in the command line or terminal, so the player can type commands into the game similar to text-based games like Colossal Cave Adventure or Zork. Audio format support: During our research, we were looking for a library that supported the FLAC and OGG file formats. SDL2 has support for these audio formats. We have added more information about the Python libraries that we have researched in the notes folder. Please look at this folder for more information. Once again, thank you for sending your suggestions.