autosportlabs / RaceCapture-Pro_firmware

Firmware for RaceCapture-Pro Data Acquisition, control and Telemetry system for motorsports

GNU General Public License v3.0

65 stars 35 forks source link

MAX_VIRTUAL_CHANNELS = 30; is too small #414

Closed kosenko-max closed 8 years ago

kosenko-max commented 8 years ago

Hi. Sorry for posting here...

But I've found significant problem that is not letting me to do Evo X CAN BUS support. Standard RAX Patch logger returns 8 groups with 4 numbers in each. That's already 32 - above limit of virtual channels. Plus I have some more data outside of RAX: MAF Airflow, WGDC corrections, transmission pressure/temperature/rpm per each gearset, knockdata. Plus I'd like to read wheels speed, brake signal, steering signals from engine CAN. So I literally need much more virtual channels. And I didn't even start to use any calculated data for the primary purpose of Virtual Channels.

I don't understand from the code why limit of virtual channels on MK2 was set to 30. https://github.com/autosportlabs/RaceCapture-Pro_firmware/blob/aef0769b5539f0647a76b3a00b1774ba29a8c343/stm32_base/capabilities.h#L14

Could you move that up to 100 at least or should I do Evo X support as a custom firmware?

Thank you.

stieg commented 8 years ago

I don't recall off-hand why we set that limit either. Probably for historical reasons (MK1). We may be able to bump this up now. Just need to evaluate impact first.

stieg commented 8 years ago

Hi. Sorry for posting here...

Why? Its where it belongs. :D

kosenko-max commented 8 years ago

Is it possible to somehow get firmware with this change before release? I need to populate 74 channels http://pastebin.com/02i5Dp6L and this limit is really not letting me to do any tests.

Thank you.

stieg commented 8 years ago

Hey @kosenko-max,

We don't usually provide releases ahead of time for several reasons (stability, QC, debugging etc). But if you were so inclined you could build it yourself once the patch was in the build tree. Of course you would be on your own in terms of support (since you would not be running an official version) but I assume you already know that.

kosenko-max commented 8 years ago

It's ok, I thought maybe you have some automated build system which you can point onto 2.8.7 with DEFINE patch only and build a firmware file for testing.

I don't have build environment yet - mostly doing Lua for now to integrate Evo X CAN.

stieg commented 8 years ago

Yeah we don't have a public build system with nightly releases yet. Its on the list of things to get done.

On Fri, Feb 12, 2016 at 10:56 PM, Max notifications@github.com wrote:

It's ok, I thought maybe you have some automated build system which you can point onto 2.3.7 with DEFINE patch only and build a firmware file for testing.

I don't have build environment yet - mostly doing Lua for now to integrate Evo X CAN.

— Reply to this email directly or view it on GitHub https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183588795 .

kosenko-max commented 8 years ago

It's OK. Thank you very much. I hope I can solve memory issue now since it looks like I can't put a code that even reads that many channels from CAN. On 14 Feb 2016 13:32, "Andrew Stiegmann" notifications@github.com wrote:

Yeah we don't have a public build system with nightly releases yet. Its on the list of things to get done.

On Fri, Feb 12, 2016 at 10:56 PM, Max notifications@github.com wrote:

It's ok, I thought maybe you have some automated build system which you can point onto 2.3.7 with DEFINE patch only and build a firmware file for testing.

I don't have build environment yet - mostly doing Lua for now to integrate Evo X CAN.

— Reply to this email directly or view it on GitHub < https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183588795

.

— Reply to this email directly or view it on GitHub https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183824020 .

brentpicasso commented 8 years ago

If you'd like to share your script here we could help analyze it to see if there's a way to conserve some memory. It may also be possible to bump the memory ceiling if it's close as well.

Brent Picasso Autosport Labs Technology for Race and Street On Feb 13, 2016 9:35 PM, "Max" notifications@github.com wrote:

It's OK. Thank you very much. I hope I can solve memory issue now since it looks like I can't put a code that even reads that many channels from CAN. On 14 Feb 2016 13:32, "Andrew Stiegmann" notifications@github.com wrote:

Yeah we don't have a public build system with nightly releases yet. Its on the list of things to get done.

On Fri, Feb 12, 2016 at 10:56 PM, Max notifications@github.com wrote:

It's ok, I thought maybe you have some automated build system which you can point onto 2.3.7 with DEFINE patch only and build a firmware file for testing.

I don't have build environment yet - mostly doing Lua for now to integrate Evo X CAN.

— Reply to this email directly or view it on GitHub <

https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183588795

.

— Reply to this email directly or view it on GitHub < https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183824020

.

— Reply to this email directly or view it on GitHub https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183824111 .

kosenko-max commented 8 years ago

Sure. I think I've shared it few times http://pastebin.com/02i5Dp6L - keep updating it there. But I have to run small portions of it to test. On 14 Feb 2016 13:39, "Brent Picasso" notifications@github.com wrote:

If you'd like to share your script here we could help analyze it to see if there's a way to conserve some memory. It may also be possible to bump the memory ceiling if it's close as well.

Brent Picasso Autosport Labs Technology for Race and Street On Feb 13, 2016 9:35 PM, "Max" notifications@github.com wrote:

It's OK. Thank you very much. I hope I can solve memory issue now since it looks like I can't put a code that even reads that many channels from CAN. On 14 Feb 2016 13:32, "Andrew Stiegmann" notifications@github.com wrote:

Yeah we don't have a public build system with nightly releases yet. Its on the list of things to get done.

On Fri, Feb 12, 2016 at 10:56 PM, Max notifications@github.com wrote:

It's ok, I thought maybe you have some automated build system which you can point onto 2.3.7 with DEFINE patch only and build a firmware file for testing.

I don't have build environment yet - mostly doing Lua for now to integrate Evo X CAN.

— Reply to this email directly or view it on GitHub <

https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183588795

.

— Reply to this email directly or view it on GitHub <

https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183824020

.

— Reply to this email directly or view it on GitHub < https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183824111

.

— Reply to this email directly or view it on GitHub https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183824269 .

stieg commented 8 years ago

One thought comes to mind with your script after glancing over it. You are using 0 as the default value of the virtual channel and saying if the id == 0, then create the channels. What happens if a virtual channel id returned is 0 when you create a new channel? It will keep adding channels and will eventually memory. Try using a default value of -1 instead of 0 and changing your parameters accordingly. Or use a separate variable that tells you that you have already created the virtual channels for that particular group.

kosenko-max commented 8 years ago

I checked that - it only makes them once. Although I had a problem in a function m23Bits - it was leaking memory during bitOp and array operations. Currently has no such problem with renamings and changing to local variables.

But my main problem is not really when the code is running. I can't load this script at all - just not enough memory to parse it. And it's just 400 rows. I can try to make more functions and make smaller footprint, but I don't see how it can fit in 50kB.

On 14 February 2016 at 13:56, Andrew Stiegmann notifications@github.com wrote:

One thought comes to mind with your script after glancing over it. You are using 0 as the default value of the virtual channel and saying if the id == 0, then create the channels. What happens if a virtual channel id returned is 0 when you create a new channel? It will keep adding channels and will eventually memory. Try using a default value of -1 instead of 0 and changing your parameters accordingly.

— Reply to this email directly or view it on GitHub https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183827070 .

kosenko-max commented 8 years ago

And I am running it through minifier https://mothereff.in/lua-minifier Doesn't help much. 50kB is too small to be practical for Lua.

On 14 February 2016 at 14:00, Max Kosenko kosenko.max@gmail.com wrote:

I checked that - it only makes them once. Although I had a problem in a function m23Bits - it was leaking memory during bitOp and array operations. Currently has no such problem with renamings and changing to local variables.

But my main problem is not really when the code is running. I can't load this script at all - just not enough memory to parse it. And it's just 400 rows. I can try to make more functions and make smaller footprint, but I don't see how it can fit in 50kB.

On 14 February 2016 at 13:56, Andrew Stiegmann notifications@github.com wrote:

One thought comes to mind with your script after glancing over it. You are using 0 as the default value of the virtual channel and saying if the id == 0, then create the channels. What happens if a virtual channel id returned is 0 when you create a new channel? It will keep adding channels and will eventually memory. Try using a default value of -1 instead of 0 and changing your parameters accordingly.

— Reply to this email directly or view it on GitHub https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183827070 .

stieg commented 8 years ago

Sounds like you are hitting the parsing limit of our LUA run time. Beware that Lua will parse methods even if they are not invoked (See https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/411). There is a fix coming for that in the next release that is already in r/2.8.8. If you are having load issues try removing the unused methods to see if that helps.

RE the memory... frankly you aren't using much at all. You are storing several IDs, but not anywhere close to 50K worth. It is possible that something is leaking memory and is causing this issue. I would do the following:

Prevent the re-invocation of addChannel by setting those defaults to -1 or using a separate variable. I know that it shouldn't cause an issue, but lets eliminate it as a possibility.
Reduce the script to only the methods that are used and see if that works. Commenting out methods completely works as the parser will ignore it.

From what I read RCP should be able to handle it. 99% of those method actions should not be doing much with memory. If there is a leak, do let us know. I assume you are figuring out all of this by viewing the RCP debug logs? If so, would you mind pasting those in too?

Also... that is an impressive script. Good work. Would be awesome to get that up on our wiki once we get past this issue :). I'm crashing for the night but do let us know what you find. Cheers.

kosenko-max commented 8 years ago

Thank you. Yes, I'm exactly in that problem - Lua can't fit script DOM into memory. My original plan was to collect all data at different rate plus use some simple settings that would allow to quickly switch to transmission or engine diagnostics. I was planning to do DTC retrieval and alerting as well as ROM flashing if you make some IO functions.

But now its obvious that I can't do it. Not in Lua at least. Do you see if I can maybe extend hardware memory through I2C or something? How much total do you have there? It looks like for whole thing to work I would need like 0.5Mb of memory. I don't see a processor speed problem yet.

So I would double check again that there is no reinvocations happening.

But I can't really comment unused code. Because its all there for use. It was commented for the sake of the testing.

I was thinking that once it's done it would be good to put in your github so that people can evolve it. There are a lot of awesome things could be done in future. Think plugins that you can choose in app. Think GUI management for app through the same scripts.

But tiny memory could kill a whole thing. If you sure that 50kb limit for MK2 is reasonable - I only see it possible by some memory extension module or by MK3 with some modern amount like 0.5-1.0 Gb. What do you think? On 14 Feb 2016 14:21, "Andrew Stiegmann" notifications@github.com wrote:

Sounds like you are hitting the parsing limit of our LUA run time. Beware that Lua will parse methods even if they are not invoked (See #411 https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/411). There is a fix coming for that in the next release that is already in r/ 2.8.8. If you are having load issues try removing the unused methods to see if that helps.

RE the memory... frankly you aren't using much at all. You are storing several IDs, but not anywhere close to 50K worth. It is possible that something is leaking memory and is causing this issue. I would do the following:

Prevent the re-invocation of addChannel by setting those defaults to -1 or using a separate variable. I know that it shouldn't cause an issue, but lets eliminate it as a possibility.

Reduce the script to only the methods that are used and see if that works. Commenting out methods completely works as the parser will ignore it.

From what I read RCP should be able to handle it. 99% of those method actions should not be doing much with memory. If there is a leak, do let us know. I assume you are figuring out all of this by viewing the RCP debug logs? If so, would you mind pasting those in too?

Also... that is an impressive script. Good work. Would be awesome to get that up on our wiki once we get past this issue :). I'm crashing for the night but do let us know what you find. Cheers.

— Reply to this email directly or view it on GitHub https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183829253 .

kosenko-max commented 8 years ago

Double-checked that initialization only happens once and that out-of-memory during parsing stage - non of the code can start. Here is example.

Small part of code works.

But if I'm adding just few more channels it says:

lua: Completed updating LUA. Flashing... win! lua: Initializing... lua: Loading lua script (len = 8422): [lua] Memory ceiling hit: 51524 > 51200 ERROR! lua: startup script error: (not enough memory)

' local chBarometer,chMAP,chBoost,chWGDC,chMAFVolt = 0,0,0,0,0 local chMIVInTar,chMIVInAct,chMIVExTar,chMIVExAct = 0,0,0,0 local chMIVInErr,chMIVExErr,chMIVActOver,chMIVTarOver = 0,0,0,0 local chTPS,chAPP,chMAFAirTemp,chWGDCCorr = 0,0,0,0 local chSpeed,chECUVolt,chWaterTemp,chInAirTemp,chInAirDiff = 0,0,0,0,0 local chLoadMAF,chLoadMAP,chLoadI_MAP,chLoadChosen = 0,0,0,0 local chFrontO2 = 0

--RAX D d chBarometer = addChannel("Barometer", 100, 2, 6.5, 15.6, "PSIA") chMAP = addChannel("MAP", 100, 2, 0, 58, "PSIA") chBoost = addChannel("Boost", 100, 1, -14.5, 43.5, "PSI") chWGDC = addChannel("WGDC", 100, 1, 0, 100, "%") chMAFVolt = addChannel("MAFVolt", 100, 2, 0, 5, "V")

--RAX E d
chMIVInTar = addChannel("MIVInTar", 100, 1, -40, 40, "deg")
chMIVInAct = addChannel("MIVInAct", 100, 1, -40, 40, "deg")
chMIVExTar = addChannel("MIVExTar", 100, 1, -40, 40, "deg")
chMIVExAct = addChannel("MIVExAct", 100, 1, -40, 40, "deg")
chMIVInErr = addChannel("MIVInErr", 100, 1, -80, 80, "deg")
chMIVExErr = addChannel("MIVExErr", 100, 1, -80, 80, "deg")
chMIVActOver = addChannel("MIVInAct", 100, 1, -80, 80, "deg")
chMIVTarOver = addChannel("MIVExTar", 100, 1, -80, 80, "deg")

--RAX F d
chTPS = addChannel("TPS", 100, 1, 0, 100, "%")
chAPP = addChannel("APP", 100, 1, 0, 100, "%")
chMAFAirTemp = addChannel("MAFAirTemp", 100, 0, -40, 100, "C")
chWGDCCorr = addChannel("WGDCCorr", 100, 1, -64, 64, "%")

--RAX G d
chSpeed = addChannel("Speed", 100, 0, 0, 300, "Km/h")
chECUVolt = addChannel("ECUVolt", 100, 2, 0, 18.6, "V")
chWaterTemp = addChannel("WaterTemp", 100, 0, -40, 150, "C")
chInAirTemp = addChannel("InAirTemp", 100, 0, -40, 210, "C")
chInAirDiff = addChannel("InAirDiff", 100, 0, -100, 100, "C")

--RAX H d
chLoadMAF = addChannel("LoadMAF", 100, 0, 0, 500, "Load")
chLoadMAP = addChannel("LoadMAP", 100, 0, 0, 500, "Load")
chLoadI_MAP = addChannel("LoadI_MAP", 100, 0, 0, 500, "Load")
chLoadChosen = addChannel("LoadChosen", 100, 0, 0, 500, "Load")

chFrontO2 = addChannel("FrontO2", 100, 2, 0, 1, "V")

stieg commented 8 years ago

How much total do you have there?

192K total on the stm32f4 chip.

It looks like for whole thing to work I would need like 0.5Mb of memory.

How are you coming to that conclusion exactly?

If you sure that 50kb limit for MK2 is reasonable - I only see it possible by some memory extension module or by MK3 with some modern amount like 0.5-1.0 Gb. What do you think?

50K was chosen as the upper limit for Lua since during internal testing I was able to show that if the run time got much beyond that it would destabilize the system to the point where the memory allocator would run out of RAM and cause a hard fault. Hence the 50K limit (would rather have Lua crash than the system). That limit was found back around the 2.8.4 time frame.

More memory would solve the problem here of course, but obviously that requires a full re-design. Whenever the next major revision of RCP happens, there will be many many significant changes, one of which will be moving to a platform that has 1 - 2 orders of magnitude more RAM.

I'm still personally of the opinion that limited RAM is not the culprit here. You simply are not using enough variables for that to factor in. If you were using something like 2000+ variables, then there would be a potential issue; but that is not the case. Just gotta figure out what is chewing up all that RAM and why.

kosenko-max commented 8 years ago

Oh, 192kb. I guess for me the only option is to step down to custom firmware. I'm certainly not using anything even remotely that consumptive, but Lua is interpreter and every line of code takes quite a lot of memory. You can see that Lua without code takes 13kB already. Small code and it bumps to 30. I come to conclusion about 0.5-1.0Mb pure on watching how fast it eats memory after parsing. If I'm doing something like DTC management, calculations over data, trends calculations, support for different Evo X versions - that would easily go to such range. For now I can't even log what I was logging for the past 2 years. And I would appreciate if you can help achieving this.

I understand that with 192kb total - 50kb for Lua makes sense. I'm not sure why you didn't made MK2 with external DRAM support. As far as I understand it's STM32F407 so it doesn't have any SPI SDRAM support, so no much luck to make any externally connected module. Or am I wrong?

I would suggest to think much larger RAM if you want to see developers do plugins on Lua. Raspberry Zero is $5 1Ghz 0.5Gb RAM PCB. I understand that you probably didn't expected anyone actually start to use Lua above few channels. But there is such option now and backward compatibility is what can make troubles here

To be honest I'm confused now about what I should do next. For a long time I was thinking about making own open source platform for amateur car racing. With modern cheap hardware it could really pick up if it has friendly flexible environment for developers. And I am extremely excited that I found you through our fellow forum resident. You have open source software, made public process of development, accepting pulls. And general approach looks very promising. There are dozens of companies on the market in the loggers field. Some already for decades. But they ignore amateurs field by providing something that barely usable or quickly becomes too expensive with very limited extensibility. What you do is the most developer friendly. And now this bummer with a RAM. You already have a headache supporting MK1 and now jumping to MK3 so soon... Plus non pro version, I guess, already in the last stages and there are hundreds of guys waiting for their stuff. So I don't have high hopes here. But as a consumer products factory owner, CTO and ex software developer I suggest you to consider this opportunity - make platform much more developer friendly. 192Kb is a very serious limitation. And if it's not too late - consider to add some cheap SDRAM to a non-pro version.

Back to the question.

If you can try to find a way how to force Lua to make code much more compact - that could solve a problem for a time being. Again, I'm 100% certain - code itself is not a problem - it's not running. Code DOM can't fit into 50kB. Is there any way to make Lua be more RAM efficient - you might know better. Something like that? http://www.eluaproject.net/doc/v0.9/en_arch_ltr.html or even LuaJit http://luajit.org/luajit.html - it has ARM v7 support for quite a time already.
Next step could be to put some higher level methods into firmware. Like byte array bit access or CAN request-response methods I have in Lua.
Next step could be to extend CAN mapping to support requests and multi-packet responses with bit shift access and more advanced algebra.
And last step is to check how to initialize 100-200 channels without draining memory so fast. Just variable declarations and 20 lines of code calling addChannel eat another 20kB at parsing? Something is not very efficient there.

If you look at my code - I do nothing more than a standard CAN messaging, bit array, channels setup and numbers conversion. You're absolutely right - that doesn't worth 50kB at all. And steps above is how you can help make it possible for MK2 (maybe even MK1).

In a meantime I would really appreciate if you can make some virtual machine with developer environment ready to use for firmware. I'm not from the Unix field at all, so idea of spending another week to learn how to setup it is really making me sad. And besides - last week was my holidays week, so I had a time to do script without any IDE just to find out that it doesn't fit into memory. Even 50% of it... Next week I'm going to have full work schedule setup with only 1 hour a day. And spending few weeks on a IDE setup for firmware is demotivating idea. If I have some ready to use environment with simple instructions - I can do useful code like CAN messaging, DTC support, bitarrays, logs compacter and stuff like that.

Sorry, my comments in the Issues look like letters. But I don't know your emails and I really want to help make platform popular.

Thanks.

On 14 February 2016 at 23:57, Andrew Stiegmann notifications@github.com wrote:

How much total do you have there?

192K total on the stm32f4 chip.

It looks like for whole thing to work I would need like 0.5Mb of memory.

How are you coming to that conclusion exactly?

If you sure that 50kb limit for MK2 is reasonable - I only see it possible by some memory extension module or by MK3 with some modern amount like 0.5-1.0 Gb. What do you think?

50K was chosen as the upper limit for Lua since during internal testing I was able to show that if the run time got much beyond that it would destabilize the system to the point where the memory allocator would run out of RAM and cause a hard fault. Hence the 50K limit (would rather have Lua crash than the system). That limit was found back around the 2.8.4 time frame.

More memory would solve the problem here of course, but obviously that requires a full re-design. Whenever the next major revision of RCP happens, there will be many many significant changes, one of which will be moving to a platform that has 1 - 2 orders of magnitude more RAM.

I'm still personally of the opinion that limited RAM is not the culprit here. You simply are not using enough variables for that to factor in. If you were using something like 2000+ variables, then there would be a potential issue; but that is not the case. Just gotta figure out what is chewing up all that RAM and why.

— Reply to this email directly or view it on GitHub https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-183907699 .

stieg commented 8 years ago

I understand that with 192kb total - 50kb for Lua makes sense. I'm not sure why you didn't made MK2 with external DRAM support. As far as I understand it's STM32F407 so it doesn't have any SPI SDRAM support, so no much luck to make any externally connected module. Or am I wrong?

This particular chip series does support external SRAM, but I imagine @brentpicasso chose not to use it due to the extra pins that would be consumed (along with reduced memory speed of course). Also, one more component that was not thought as needed at the time. Coming from MK1 where the on board ram was limited to 64K, 192K was quite an improvement.

Raspberry Zero is $5 1Ghz 0.5Gb RAM PCB

Yep. Also came out only a few months ago and it quite hard to get a hold of one these days. Believe me... we are noticing these cheap new chips and are looking (along with the 10,000 other projects we would like to do).

I understand that you probably didn't expected anyone actually start to use Lua above few channels. But there is such option now and backward compatibility is what can make troubles here.

We expected up to 30 channels, and ideally more when we could get to testing and approving expansion. Honestly the fact that this script of yours is crashing the LUA runtime is suspicious. LUA is designed to be very light weight, so something tells me there is possibly an issue here.

So I don't have high hopes here.

Well don't give up so easily. Like I have said... I suspect a memory leak or some kind of bug in there to be the root cause of this issue. Since you have SWE experience I suggest you setup a build environment and play with the firmware a bit. Checkout the project README on the page, setups for a Linux machine are there. If you are on a Mac, I put together a vagrant image a while back. Probably needs an upgrade, but it something you can work with.

Also... don't forget that direct firmware CAN support is in the pipeline. See issue #172 for more info. That will probably replace the work you are doing in LUA, so I wouldn't be so quick to fly the white flag.

And if it's not too late - consider to add some cheap SDRAM to a non-pro version.

RaceCapture itself doesn't support LUA (we will be releasing it with C based CAN support). So no worries there.

Is there any way to make Lua be more RAM efficient - you might know better.

There are ways, but there are often dragons there too. Sometimes enabling those ways causes issues (like the LUA math libs all of a sudden stop working).

Next step could be to put some higher level methods into firmware. Like byte array bit access or CAN request-response methods I have in Lua.

Yep. Right track here. IMHO what we need to do is setup a periodic CAN message sender that will send messages at certain intervals (like your PIDs). Then you simply just listen for the incoming addresses and parse appropriately.

Next step could be to extend CAN mapping to support requests and multi-packet responses with bit shift access and more advanced algebra.

Again... see #172. Currently we aren't focused on handling more than 8 byte responses. That may come down the road as needed. We also do accept patches. See the contributing doc.

In a meantime I would really appreciate if you can make some virtual machine with developer environment ready to use for firmware. I'm not from the Unix field at all, so idea of spending another week to learn how to setup it is really making me sad.

We will do what we can, but do understand we have other promises to deliver on as well. I understand that learning a new environment is time consuming too... but software is never an easy field. Always a battle until you (hopefully) prevail and enjoy that moment of bliss and glory at the top of the mountain... only to then start all over and climb another.

kosenko-max commented 8 years ago

Well, I'm not giving up yet. I don't have high hopes to get MK3 or memory upgrade soon though. So full scale plugins are not nearest future on a RCP. I can try to use Vagrant. I don't know if it's possible to setup environment in Cloud 9 - that would be ideal case.

172 is not planned to have any requests as I got it from discussion there. It could do all I wrote at this point with proper requests, multi-page responses and bit shift logic, though. But I understand that's not a plan.

I much less worry what MK2 can do than what platform can do. RaceCapture being cheaper is more attractive for building community, but then again - even less memory (as I understand), no Lua, no GPS external antenna. If that was me - I would go for powerful inexpensive core block with robust plugin system and extensions for more IO. Something future proof that attracts developers. And make most business out of extended functionality plans on Podium. You can't possibly cover everything by "just fit for basic functionality" hardware or expect to develop everything by few super efficient developers who also need to manage whole business. But currently what you did, even if it's open source, documented, git based - it's not even Arduino level simplicity for developers. Whatever my previous dev experience was - toolchain setup and learning is a significant time taker. And even if I would do that - you're not really planning that people going to be growing by doing all kind of custom firmware - it's a minefield.

I don't know if you're using eLua already, but it looks to me that it's especially setup for those type of tiny controllers. If you're not - that's what you can check maybe?

Problem with message sender separated from CAN mapping is that there is no way sometimes to associate response with any specific answer - it's just doesn't have a descriptor about request. So CAN mapping would have no way to understand what was that question about. Address is the same, no PID in the answer.

My Lua script is making that ISO multi message CAN protocol to some extent. I can extend the code, no problem. But if CAN mapper has no request-response sequence option - it's useless for me.

I have a Mac, so if you point me where to take Vagrant image - I would check how to use it. In readme it's written that it will be done some day.

brentpicasso commented 8 years ago

167 is the feature that does OBD-II style mapping. As @stieg indicated previously, it will use similar mapping scheme behind the scenes as #172, but will offer the ability to map PID requests to response messages. The user interface will an extended version of what we have for OBD-II configuration, but still simplified.

We hadn't included the requirement for multi-message CAN protocol, but can investigate including it.

Would it be possible to update the ECU firmware to simply broadcast the desired data without requiring the PID-style request-reply scheme? This would emulate the approach of virtually every after-market ECU, on top of making the entire process more efficient.

kosenko-max commented 8 years ago

Evo X has engine harness ECU tap point that does broadcasting. But it has very small limited set of driving data. There is nothing that helps tuning, diagnostics or even temperature control. And it's not possible to change it

some data is not even from ECU - there are TCU, AWD and some other sources of data that all have to be request response based. I'm still considering to get that data as well, but in such case both CAN channels would be taken by ECU.

And again, main problem is memory limit. When you said Lua is for flexibility of doing what you want - I agree with that. But if it limits me to 200 lines of code and I have to fit there all CAN messaging logic and bit array helpers - that is just not flying at all.

I'm trying to make simple, but robust plug and play system for Evo X owners. Custom firmware is not going to work for that - how people going to be getting firmware updates on a base firmware?

My only hope is that if you can help me minimize Lua code memory requirements by moving some helper code into master firmware and optimizing Lua environment so that it can run more lines (eLua, LTR, JIT, embedded precompiler and minimizer, Lua templates library in the app) or making CAN mapper more robust to support logging without Lua.

At this moment for me its not even clear if my changes to firmware would be accepted in a master branch since long term code evolution strategy is unknown.

Could I ask about increase of limit of virtual channels and help investigating memory consumption of Lua with Vagrant images for dev env? On 16 Feb 2016 07:53, "Brent Picasso" notifications@github.com wrote:

167

https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/167 is the feature that does OBD-II style mapping. As @stieg https://github.com/stieg indicated previously, it will use similar mapping scheme behind the scenes as #172 https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/172, but will offer the ability to map PID requests to response messages. The user interface will an extended version of what we have for OBD-II configuration, but still simplifie d.

We hadn't included the requirement for multi-message CAN protocol, but can investigate including it.

Would it be possible to update the ECU firmware to simply broadcast the desired data without requiring the PID-style request-reply scheme? This would emulate the approach of virtually every after-market ECU, on top of making the entire process more efficient.

— Reply to this email directly or view it on GitHub https://github.com/autosportlabs/RaceCapture-Pro_firmware/issues/414#issuecomment-184445524 .

stieg commented 8 years ago

Patch tracked in #440

stieg commented 8 years ago

Merged