Closed terminar closed 3 years ago
Easy part first: What -O
option did you use for compilation? At -O3
you do have to add -fno-builtin
because otherwise the compiler optimizes startup initialization of global variables to zero in init()
into calls to memset(). Alternately, you can link with -lc
to get GCC's memset()
.
Hard part: The buck50 code, in particular the papoon_usb library (included in the buck50 repo), is failing USB device enumeration. I assume that because you're compiling on macOS you're connecting buck50 to the same. I don't have any Apple machines to test on, and this is the first report of any kind I've had, so unsure if the problem is in papoon_usb or on the Mac side.
Can you try compiling a USB-CDC example from ST's "Cube F1" distribution and running on your bluepill hardware to see if that works? (I'm sure all normal USB devices work on the Mac.) You will have to do some minor porting from the code's dependency on an ST evaluation board, and deal with the "Cube"/HAL build system if you haven't done so already.
If that works, papoon_usb is the problem. I do know that it doesn't fully respond to USB-CDC requests from the host, for example baud rate setting. (This is part of the insanity of USB in general and USB-CDC in particular -- there isn't any settable "baud rate" over a USB cable, but USB was designed 20+ years ago when USB-to-RS232/modem converters were likely a primary use case. Instead of layering baud rate, word length, stop bits, etc. control over a generic bytestream communication API they were baked into CDC, and because that's the standard, it never changed. I'm guilty of this too: I used CDC instead of my own custom USB class so buck50 could run without a userspace USB driver in the host buck50.py code.)
If you want to get your hands dirty, look at device_class_setup()
in usb_dev_cdc_acm.cxx. If macOS is hanging for the reasons I stated above, the fix would either be there, in the currently null UsbDev::set_configuration()
and/or UsbDev::set_interface()
methods, or in the worst case scenario, additions to UsbDev::device_request()
in usb_dev.cxx. The general idea would be to convince the macOS driver that you'd listened to its "set" commands, and return reasonable values if it does a "get" afterwards.
Like I said, this is all a tremendous mess, and particularly because I don't have any way to test it I'm not likely to have time right now to address the issue. If in truth this is the problem (it could be something completely different) and you'd want to work on it, I'd welcome any non-obtrusive patches (after of course testing to make sure they didn't break the existing Linux usage).
Hope some of this has at least been interesting, and sorry that I can't be of more help.
One more thing, to quote a famous Apple personality ;) -- if you don't know already, when you're debugging as you obviously are, you cannot simply restart USB device code such as on the bluepill when you want to test a change. You need to physically unplug from the Mac's USB port or any USB hub in between and re-plug to force re-enumeration of the device. I do have some open-source code that I had to modify from the original author's version which can force re-enumeration via software, but it's very experimental, temperamental, Linux-specific, and relies on having specific hub hardware capabilities.
First of all, thanks for the detailed explanation and your time!
Easy part first: What
-O
option did you use for compilation? Default from build/Makefile which seems to be -O2 (OPTIMIZE_FLAG). Full line as example:arm-none-eabi-g++ -c -std=c++17 -Wall -Wextra -g3 -mthumb -mcpu=cortex-m3 -fno-exceptions -fno-unwind-tables -fno-builtin -I. -I./init -I./include/arm -I./include/st/32f -I./include/thanks4opensource -I./util/stm32f10_12357xx -O2 -DUSB_DEV_FLASH_WAIT_STATES=2 -DINLINE_DECL=inline -DINLINE_ATTR='__attribute__((always_inline))' -fno-threadsafe-statics src/buck50.cxx -o buck50.o
At
-O3
you do have to add-fno-builtin
because otherwise the compiler optimizes startup initialization of global variables to zero ininit()
into calls to memset(). Alternately, you can link with-lc
to get GCC'smemset()
.
I'll stick with -fno-builtin at the moment.
I assume that because you're compiling on macOS you're connecting buck50 to the same.
Correct. But even if I don't think often about it I remember I have Windows and FreeBSD systems around me. Tried it on the windows system. The Buck50-bluepill is working on windows and the virtual com port is available. The same device doesn't appear on my macOS Catalina. I can't find the time today to test the CubeMX F1 stuff but I feel certain that this is maybe really a bug in the papoon_usb. Today I tried
I don't have any Apple machines to test on, and this is the first report of any kind I've had, so unsure if the problem is in papoon_usb or on the Mac side.
I can assist in testing (and maybe find a solution) but I tend to say that generally it's a papoon_usb problem because I have many gadgets and helper in use which are working without any problem.
I'm guilty of this too: I used CDC instead of my own custom USB class so buck50 could run without a userspace USB driver in the host buck50.py code.)
I appreciate that! Custom driver stuff on macOS (and even on Windows with that signing stuff) is getting really annoying.
If you want to get your hands dirty, look at
device_class_setup()
in [usb_dev_cdc_acm.cxx]...
Getting hands dirty isn't the problem. I like the general idea of Buck50 because I love(d) my bus pirate but I am not a friend of PIC. Also it's a little bit outdated and there are so many (cheap) usable stm32 devices with much more performance and possibilities... That's why I wanted to tinker with buck50. It's reasonable to use a host tool like buck50.py and the hardware part, also great idea.
But (and I don't want to be rude or doing too much critics!!) - the code of buck50.cxx is - interesting. It's great to have the comments and generally it looks readable. Thank you for using empty lines! Many programmers are having great fear of these little things. But still the code feels so compressed into that one c++ file with 4040 lines. Even if it's C++ the readability is limited for my eyes (but that's subjective, of course). It just feels amalgamated into one file.
If in truth this is the problem (it could be something completely different) and you'd want to work on it, I'd welcome any non-obtrusive patches (after of course testing to make sure they didn't break the existing Linux usage).
I think "the world" needs something like Buck50 as sort of successor to devices like the "bus pirate". I think, generally the stm32 devices are perfect for this. I am also very open to contribute but "non-obtrusive" is of course a nice, legit word combination which can be interpreted as: "Bugfix is OK, refactoring is not". What exactly is your interpretation of "non-obtrusive"?
Hope some of this has at least been interesting, and sorry that I can't be of more help.
Yep, of course. It's just a question how we want to go further. When I start to get "my hands dirty" I have some own goals for the future, at least
Generally that first of all means open questions:
You need to physically unplug from the Mac's USB port or any USB hub in between and re-plug to force re-enumeration of the device.
Yep. I am using an USB hub with switches. And I'm detaching my cables most of the time as "reboot" - but thanks for clarification.
I do have some open-source code that I had to modify from the original author's version which can force re-enumeration via software, but it's very experimental, temperamental, Linux-specific, and relies on having specific hub hardware capabilities.
Too specific for me ;) I think detaching cables is reasonable.
I'm really interested what your plans are for buck50 in the future - hope we can find the same goals!
My own "first of all": Thanks for the interest and input, @terminar. Here are some of my thoughts ...
Tried it on the windows system. The Buck50-bluepill is working on windows and the virtual com port is available.
That's very interesting, and although I don't want to imply in any way that it "proves" that papoon_usb is fully USB spec compliant, it does seem to indicate that macOS is doing something that Linux and Windows are not. Like I said, the USB standards are a convoluted mess and it's not surprising that different host implementations behave differently.
blackmagicprobe -> working => ... desk-viking -> working => ...
I took a quick look at the code (thanks for the concise links) and they basically seem to be handling get/set_line_coding and _control_line_state, like papoon is. I'm sure there are differences, though.
As a side note, I've looked at the libopencm3 codebase before and find it both very hard to follow and the API very hard to use. Especially compared to my regbits. But that's a proud father bragging about his child compared to other kids. ;)
Also, re: "desk-viking": Not surprising that others have done this. I had known about the "bus pirate" but frankly had completely forgotten about it by the time I started buck50. (Is "bus_pirate_ng" -- you didn't include a link -- the code for the commercial "bus pirate" product?) As I note in the README.md, buck50 started out just as a logic analyzer but kept growing as I realized I could add more and more features. Also about regbits: What is the binary size of desk-viking and bus_pirate_ng (not the size of the .elf file, but the actual running ARM text+bss+data) compared to the same for buck50 (which does much more)? And one more: Regardless the fact that it's based on opencm3 and I have some objections to its design (hardware device descriptions in the binary instead of in host-based configuration files like OpenOCD), I have great respect for Black Magic Probe and in fact own one in hardware that I purchased from "1 Bit Squared".
I can assist in testing (and maybe find a solution) but I tend to say that generally it's a papoon_usb problem because I have many gadgets and helper in use which are working without any problem.
As I said, I'm sure macOS works with most USB devices.
I gave a Mac-based friend a buck50, but as he's a professional with thousands of dollars of logic analyzers and other test equipment I don't think he ever tried it. I'll have to beg him to do so as another data point, but again, I don't think I (and certainly not he) will be able to take the lead in getting papoon working on macOS.
I appreciate that! Custom driver stuff on macOS (and even on Windows with that signing stuff) is getting really annoying.
The "driver" would have been libusb in user, not kernel, space, so as long as macOS/Windows allowed access to the raw device it shouldn't have been a problem. Would also have allowed buck50.py to run on Windows as libusb is cross-platform.
But I know what you're saying about custom drivers. And I still think CDC-ACM was the right way to go.
there are so many (cheap) usable stm32 devices with much more performance and possibilities...
That's why I made buck50. The hardware was sitting there, so why not use it?
the code of buck50.cxx is - interesting. ... It just feels amalgamated into one file. ... I am also very open to contribute but "non-obtrusive" is of course a nice, legit word combination which can be interpreted as: "Bugfix is OK, refactoring is not". What exactly is your interpretation of "non-obtrusive"?
Good questions. Let me try to clarify ...
I know my coding style is somewhat out of the ordinary. I'm OK with that, and I'm willing to defend its merits to anyone who's interested. By "non-obtrusive" I basically meant I'd prefer not to get something like a one-line change wrapped in, "I hate your indentation style so I fixed it and also changed all the spaces to tabs and ..." and the diff is then the whole file. I believe, and try to follow, the idea that when submitting patches to someone else's code one should follow its style as closely as possible, or at worst only change style in the new code while leaving the rest alone.
As far as the larger issues of:
split buck50.cxx to be more readable and maybe more modular ... maybe think about some sort of plugins/modularity
those are ... larger issues. Defending my choices, I have no problem with large files and/or large functions. (I know this puts my into a small minority.) Any modern system/editor can handle files of any size. Contrast this with breaking everything into small, separate files. Unless the splitting is very well designed and named, it's obvious that the writer is using some kind of IDE which makes it easy to jump to an arbitrary file/directory/project to find a piece of code. My position is that code readability should not depend on using those kinds of development environments.
Likewise with long functions. I hate spaghetti code, but I find no advantage in breaking up a long, linear function with commented sections of, "First we do this", "Next this needs to get set up" into a short function which calls first_we_do_this()
, next_this_needs_to_get_set_up()
and so on. Once again, that requires jumping around in a single large file at best, and to different files using an IDE at the worst. (Of course if a "do this stuff" section is reused or called multiple times from different places in the code it gets broken out into its own do_this_stuff()
function.)
Once again, I realize all this goes against what's being taught in every Computer Science curriculum these days, but I'm fine with that. ;)
So getting back to the practicalities, you are of course free to do what you want, as long as (of course) you respect the terms of the GPL license. If you do end up branching off your own fork of buck50 I'd also appreciate you crediting the original, as I will you if I take any of your contributions back into my repository (see my release notes in my various projects for proof that I do this). If you find it necessary to do a major refactoring of the code, users will decide which version they want to use, and/or I could extract the substantive changes back into my codebase, and/or I could hand off the whole project to you and let you maintain it. ;) I have plenty of other things to work on (but I am interested in finding out what the papoon_usb problem is).
In terms of a port to STM32F401/411, "black pill", etc, the biggest problem there is that they contain the "new" ST USB-OTG peripheral which is completely different and incompatible with papoon_usb as now written for the "old" device-only one in the STMF103. This would require a complete rewrite (or using opencm3, etc) which is something I want to do at some point for use with higher-end ST MCUs. But note I was warned by a knowledgeable person on the ST forum that writing a driver for the "new" is even worse than for the "old", and papoon took almost six month to write due to the obfuscated ST code I had to use as an example because the reference manual completely lacks the detail needed to create a driver from scratch.
The increased speed (100MHz vs 72, right?) of the 401/411 chips isn't that much (and the only part of buck50 that's CPU speed dependent is the logic analyzer) although the increased RAM would be nice. But I'll tell you that for every request for a 401/411 port I've seen there's been two criticizing, "Why did you spend time writing this crappy 6+ MHz logic analyzer? Don't you know that you can buy Chinese Cypress FX2 clone boards for $6 that run at 20 MHz with infinite host-based memory buffers?"
They have a valid point. As I said, I wrote buck50 because the hardware "was there". (I might not have if I'd known in advance how large the project would grow and how much time it would eat up.) It's up to you where you want to go with the other chips. I can likely offer advice and ideas, but probably not much help in actual code development.
Thanks for reading all this, and I'm interested in your reactions to it.
(Is "bus_pirate_ng" -- you didn't include a link -- the code for the commercial "bus pirate" product?)
Hm, yes and no. The bus pirate is/was based base on a PIC18F2550 and later on, PCI24FJ64GA002. The original code is somewhere else, the "bus pirate ng" was the effort to do the same with stm32 but the project seems to be dead.
As I note in the README.md, buck50 started out just as a logic analyzer but kept growing as I realized I could add more and more features.
Understood, different starting points and motivation.
Also about regbits: What is the binary size of desk-viking and bus_pirate_ng (not the size of the .elf file, but the actual running ARM text+bss+data) compared to the same for buck50 (which does much more)?
Regbits may be more efficient. Compared to desk-viking Buck50 it does of course more. Compared to bus pirate, that depends on details what is meant with "does much more". Desk viking => ~20k Bus Pirate NG => ~50k
And one more: Regardless the fact that it's based on opencm3 and I have some objections to its design (hardware device descriptions in the binary instead of in host-based configuration files like OpenOCD), I have great respect for Black Magic Probe and in fact own one in hardware that I purchased from "1 Bit Squared".
Same with me, but it's working better than expected.
The "driver" would have been libusb in user, not kernel, space, so as long as macOS/Windows allowed access to the raw device it shouldn't have been a problem. Would also have allowed buck50.py to run on Windows as libusb is cross-platform.
In my experience using libusb is hell on earth (on windows), not sure on macOS but I have in mind that I also had problems. I can't remember what device I used on Windows but I really got into big problems with driver signing and so on. There are tools like Zadig for it but even then, it was a mess.
But I know what you're saying about custom drivers. And I still think CDC-ACM was the right way to go.
Yes!
By "non-obtrusive" I basically meant I'd prefer not to get something like a one-line change wrapped in, "I hate your indentation style so I fixed it and also changed all the spaces to tabs and ..." and the diff is then the whole file. I believe, and try to follow, the idea that when submitting patches to someone else's code one should follow its style as closely as possible, or at worst only change style in the new code while leaving the rest alone.
Same with me. I don't have a problem with different formatting. And also I don't like religious likes debates about formatting code, how brackets should be set and so on. A project with different members will find its code style itself and also imho it depends what language is used. But even then, if it's a personal project all that doesn't matter - as long it is readable. That's also the reason why I am not a big fan of Python because it dictates me how I have to format my code - or it does not work as expected. The idea itself is great and OK but that shouldn't be linked together (but that's another discussion and again, simply personal).
those are ... larger issues. Defending my choices, I have no problem with large files and/or large functions. (I know this puts my into a small minority.)
I have no problem with large functions and large files. I have a problem with density and non-visible or missing layering/project structure, missing "grouping". Imho (!) buck50.cxx just makes it hard to jump into the code due to the density and missing grouping/mixed code. I can't focus just on "protocols" or just on the "core" at first.
Any modern system/editor can handle files of any size.
That's not my point (and even the non-modern midnight commander viewer was able to load/stream gigabytes of files in 1996). It's a matter of readability and grouping.
Contrast this with breaking everything into small, separate files.
I don't have a problem where several classes are contained in one source code file, if they have things in common. If they are logically grouped.
Unless the splitting is very well designed and named
I assumed that.
My position is that code readability should not depend on using those kinds of development environments.
But you are forcing to :) I HAVE to use an IDE to get a view of the C++ classes in buck50. I HAVE to use bookmarks in my IDE to jump faster between parts of the code. I can't just open different files to get it grouped in that files and so on ;) Maybe you have all code lines in your head but as a newcomer to the project it's complicated.
An example: L119, namespace "Haltcode" or L128, namespace SamplingMode. I have to use "find all references" in my IDE to get the logical context where that stuff belongs to. Is the sampling mode only needed to set the core frequencies of the MCU to get up and running? Is the sampling mode used in an external command to provide a frequency generator? Is this part of a user command? Is this only used for e.g. PWM?
Next: class UsbRecv. OK great, documented (but even without it's self describing that it's something with receiving USB). If I just want to look at the "user commands" I don't want to see code with USB, it doesn't matter for me, it's distracting. class Sbrk - uhm. Only by looking at the code while scrolling and the file I have no idea what that code is for. I really have to read and understand it. Next to it - union AdcLive. OK, may be part of Adc handling. And so on, and so on. Yes, you have "grouped" by sections using #if 1 - but that just group different technical stuff together in buckets. At L521 you are grouping "general utilities" (which I maybe don't need to read the next days/weeks if I will get my hands dirty). memcmp, memcpy - yes - general utilities. But in L551, why is UsbRecv::byte or ::shrt "general utility"? It's part of the UsbRecv class, not a namespace, not a utility. Maybe it's a utility used in the context of USB receiving - but that "grouping of code" obsoletes the use of C++ classes and responsibility of code. These are just loosely picked examples.
No offense, I don't want to be rude but just explain that I completely understand and accept your point of view regarding function size, formatting, ... But your position about readability causes the opposite situation for a newcomer. Putting things logical into different files with a logical grouping (like gpio.cxx, adc.cxx or whatever), "carefully" named and designed doesn't hurt but helps a lot for people who don't wrote that code. Just my two cents - of course it's your code and I appreciate your work!
Once again, I realize all this goes against what's being taught in every Computer Science curriculum these days, but I'm fine with that. ;)
As mentioned, me too. I'm not an apologist of mindless using the "clean code" term and following that blindly. I like "common sense", somewhere between 0 and 1.
So getting back to the practicalities, you are of course free to do what you want, as long as (of course) you respect the terms of the GPL license.
I was hoping not to get into a GPL - you can fork - discussion but of course I understand what your intention is. I'll don't need to go into the general situation how forks of GPL code should respect the original, I think that should be obvious.
Regarding a "fork": Imho decentralised development (using vcs like git, darcs or mercurial) has a great benefit but also killed the velocity and discussion culture of OpenSource projects. It's easier just to fork today instead of discuss what a consensus will be. I mean, take GitHub with that fancy "fork" button. But yes, it's also another word for: Get on, try to change the code, I'll take a look if it's ok for me.
I have plenty of other things to work on (but I am interested in finding out what the papoon_usb problem is).
I know that problem with "plenty of other things to work on" but the "uC swiss army knife" stuff accompanied me since a long time, that's why it interests me that much.
But note I was warned by a knowledgeable person on the ST forum that writing a driver for the "new" is even worse than for the "old", and papoon took almost six month to write due to the obfuscated ST code I had to use as an example because the reference manual completely lacks the detail needed to create a driver from scratch.
I had some nice, fast progress using the "new" CubeMX generator stuff in 2019 and the "ST USB device library" using a STM32F407vet6 as keyboard+mouse HID device and additionally with CDC-ADC that day. But yes, it still takes some time to tame that USBeast.
The increased speed (100MHz vs 72, right?) of the 401/411 chips isn't that much (and the only part of buck50 that's CPU speed dependent is the logic analyzer) although the increased RAM would be nice.
Regarding speed, the 401/411 it was just a wildcard, it also has an USB-C connector which I prefer compared to micro-usb. But just take other STM32 devices which are available around the china world, that goes up to the F7.
RAM is generally my main part. I am working with Lua since some years also in the embedded world. It would be nice to be able to write some macros and automate things on the device itself (or extend its core functionality), I did things like that with the STM32F746-DISCO. But that again is just another point of view if "host driven" or "device driven" is the correct way. I think some mixed in-between will be nice.
But I'll tell you that for every request for a 401/411 port I've seen there's been two criticizing, "Why did you spend time writing this crappy 6+ MHz logic analyzer? Don't you know that you can buy Chinese Cypress FX2 clone boards for $6 that run at 20 MHz with infinite host-based memory buffers?"
Regarding logic analyzer - maybe. That's not my main motivation. I know the CY7C68013A devices. Also regarding hardware, FGPA boards are getting cheaper. I know that there are professional tools (I even own some of them). My motivation is "white smoke" and portability in the sense of mobility. I don't want to take a Saleae Pro 16 in my backpack every day if I just sometimes need a small device which can scan for I2C or bridge uart or whatever. I don't want to take around several devices if I can put one onto my key ring. And if it's broken I can throw it away and flash a new. If I've burned it into white smoke due to a failure I did, it's not that expensive. If I need a small working code platform to test something I can do that without starting "a new project". My motivation is a matter of workflow, tooling and changeability. It's not regarding price in general.
Thanks for reading all this, and I'm interested in your reactions to it.
Still, thank you for your time and I hope you don't get me wrong with the critics, it's just my subjective opinion.
Due to our current discussion I was thinking about my interests/goals (with buck50) since yesterday evening (greetings from Germany btw). At first I wanted just to tinker with it and extend it for my use cases if needed - as successor for my bus pirate workflow. Due to the problems with the papoon_usb (even if there is no bug, getting that up and running on a different mcu means work to do) I am evaluating possibilities.
When I take "STM32 MCU independence" and replace it with "platform independence" or "portability", that gets me to projects with some "RTOS" at the core with some sort of abstraction layer - I've worked with some of them in my past. Currently I am working with Zephyr which has a very very interesting design decision which I have never seen in such "RTOS" before: they adopted the Devicetree which is of course used by the Linux kernel. Zephyr supports a zoo of devices/platforms/socs. I am thinking about my motivation (and time) to go into that direction and putting some "swiss Army knife" application on top of Zephyr. I have no opinion about it yet, just thinking out load. But generally that has nothing to do with this issue anymore.
If I'll find time and the mood I'll take a look at the current papoon_usb problem on OSX and create a PR if it is working. Otherwise this issue can be seen as "currently not fixable".
Sorry to add to an already-closed issue, but I wanted to add another datapoint: on a macOS, my bluepill with buck50 is not enumerating properly. No issues on linux.
If anybody would like to fix this in the future, here's what dmesg on macOS 12.5.1 says
[1135528.904442]: 463072.874572 IOUSBHostDevice@14400000: IOUSBHostDevice::start_block_invoke: device descriptor fragment is invalid
[1135528.904708]: 463072.874842 IOUSBHostFamily::validateEndpointMaxPacketSize: USB 2.0 5.[5-8].3: endpoint 0x00 invalid wMaxPacketSize 0x0000
[1135528.905011]: 463072.875144 HS05@14400000: AppleUSBHostPort::enumerateDeviceComplete_block_invoke: enumeration failed
[1135528.905051]: 463072.875184 HS05@14400000: AppleUSBHostPort::terminateDevice: destroying 0x0000/0000/0000 (IOUSBHostDevice): enumeration failure
[1135528.905229]: 463072.875362 IOUSBHostDevice@14400000: IOUSBHostDevice::start_block_invoke: device will not be registered for matching
Thanks for the informative dmesg
dump, @tocisz. I'm not set up to work on buck50 right now, plus as stated earlier, without access to a macOS machine it would be very hard to track this down. I did a quick check of the code in build/util/stm32f10_12357xx
and it looks like the max packet size fields in the USB descriptors are being set to reasonable values, i.e. not zero. Maybe something in the macOS hardware/driver is sending down packets with a zeroed-out field and my "papoon USB" library is copying and sending it back? Just speculating -- there's a lot of that in the USB protocols and I'm not an expert on the subject. If anyone can track this down and submit a pull request I'd be happy to merge it (as long as it doesn't break Linux/Windows and isn't a rewrite/reformat of the codebase).
I tried buck50 on two classic bluepill modules, on one lctech module (also stm32f103c8t6) and on a stm32f103vet6. Connection via USB is not working (no device on host side). I tried to debug with a black magic probe (on one of these bluepill modules which is working great) and it seems that the code is trapped at
I'm using arm-none-eabi-gcc (GNU Arm Embedded Toolchain 10-2020-q4-major) 10.2.1 20201103 (release) on macOS, I had to add '-fno-builtin' to the CC_ARM_FLAGS, otherwise error occurred when compiling (couldn't link memset).
But even the shipped, precompiled buck50.elf isn't working, I'm unable to connect (via USB-CDC). Did I miss something?