Closed cjbrigato closed 5 years ago
Why did you refactor everything just when I started this work ? x) Anyway, some PR might be incoming
Blocked as of #258 and #255
You shouldn't be blocked anymore!
You got busy with your work before you could PR your stuff last autumn :) But this time the device works much better than then.
So much. Unfortunately I had forgotten why the HAS_512_FLASH did not make it and so it is now bricked. No interface available until this night so yeah, It suspended right now :(
Larger than 256kb using the flasher is bad.. until fixed, use jtag to flash 256> sized fullimages
yes and SPIFFS implies > 256k so maybe priority is to manage to flash properly 512k from bootrom. I tried briefly one day to look at it and failed miserably...
yeah.. its the swapping to bank1 after 256kb, that I failed. I dunno how large each write is...
Don't feel bad, I'll brick mine soon enough 😄 Do you have any place where the faulty swap is ? So I don't try it too...
flash.c? :)
Brace yourself people, some good soul gave me an old raspberry pi so I have a chance to unbrick my rdv4 ! The gpio's are all full of rust and the contacts are totally faulty but I'LL OVERCOME and Believe me, This PR WILL COME.
And here we go. Spiffs works, and My standalone has been modified to write it's last seen tag to /hf_colin/lasttag.bin
and to read back from it, no more arbitrary offset and low level flash operations.
Just need to also make a copy of every tag to /hf_colin/UID.bin
, then provide a couples of client and device commands to help file listing and filesystem "management" and monitoring from client and from device.
Beware : I used the SINGLETON mode of spiffs, so you can't decide dynamically of the spiffs size, it's fixed at compile time (and formatting is done by wiping the flash in between you setted up the start of spiffs + the wanted size).
Note : As it is now, getting "rid" of LF while working on spiffs is way more than enough in term of left space on internal flash (409 blocks with btaddon + hf_colin + spiffs). Until we "solve" the 512Kb case, SPIFFS being available only with non-hybrid mode don't feel strange, just as it doesn't feel strange that the midrange and long range HF antennas are "disabling" the LF too.
Edit : ha, also since I suppose everyone has a reason for this -pedantic
ism, delay is to be added to make spiffs original code C99 compliant.
You take things to the next level always, @cjbrigato !
Sorry for the delay it's becoming absolutely huge. Providing some RDV40SpiFFSSafetyLevel to file level operation including lazy_mounting and rollback to previous state, so one can live without having to manage the mounting operations.
Beware that everything works because I choose to only have ONE available file descriptor, as it is pushing less on RAM side and doesn't prevent providing abstracted copy operations. If either we chose to go along some RTOS-like operation it won't work as-is, but since I provided in such context no SPIFFS_{un}lock(fs) mecanism that move would have so much more implication that it was considered a wonthappen.
Edit: SPIFFS original code is now C99 compliant btw and beware that string.c/string.h is growing quite a bit as operation on an FS are becoming more complex. There will be PLENTY of room for optimization-whores hanging around.
Now providing some read_as_symlink function to avoid making copies when a symlink would have work
As an exemple, now every tag are saved in hf_colin/uid.bin then hf_colin/lastag.bin is truncated and filled with path to actual last tag.
A rdv40_spiffs_read_as_symlink
then provide a short hand to actually read the real last tag file when providing the last tag file as argument. Note than the abstraction makes the size argument of such function to be the size needed for destination file, but not symlink, so this is a drop-in replacement. One could even make an even more wrapped read function which would hide a read_as_symlink behavior upon condition like a .symlink
or .lnk
filename extension
If I didn't know better, I would say @cjbrigato is having way to fun! Lov'in it!
Will PR on Monday but I think some will have to take care of more client-side part. I it's ready on the device side anyway and accessible to standalone and internal operation.
ok, symlink support works, quite a lot is done for a first try so i'll just take time to benchmark/test stability and find some of the numerous race condition I may have left, clean the mess a bit, merge with recent changes, and PR the actual state on monday.
Yiha
[usb] pm3 --> mem spiffs tree
#db# Flash Memory FileSystem tree (SPIFFS)
#db# -------------------------------------
#db# /
#db# [0001] 15bytes |-- testspiffs.txt
#db# [0002] 1024bytes |-- hf_colin/mf_8e11397a.bin
#db# [0003] 32bytes |-- hf_colin/lasttag.bin
#db# [0004] 1024bytes |-- hf_colin/mf_a67f8bf5.bin
#db# [0005] 32bytes |-- hf_colin/lasttag.bin.lnk --> hf_colin/mf_a67f8bf5.bin
[usb] pm3 -->
EDIT: Resulting Rewrite of Tag save/load in my standalone :
void ReadLastTagFromFlash() {
uint16_t len = 1024;
size_t size = len;
uint8_t *mem = BigBuf_malloc(size);
rdv40_spiffs_read_as_symlink((char *)HFCOLIN_LASTTAG_SYMLINK,(uint8_t *)mem,len, RDV40_SPIFFS_SAFETY_SAFE);
emlSetMem(mem, 0, 64);
return;
}
void WriteTagToFlash(uint32_t uid, size_t size) {
uint32_t len = size;
uint8_t data[(size * (16 * 64)) / 1024];
emlGetMem(data, 0, (size * 64) / 1024);
char dest[SPIFFS_OBJ_NAME_LEN];
uint8_t buid[4];
num_to_bytes(uid,4,buid);
sprintf(dest,"hf_colin/mf_%02x%02x%02x%02x.bin",buid[0],buid[1],buid[2],buid[3]);
rdv40_spiffs_write((char *)dest,(uint8_t *)data,len, RDV40_SPIFFS_SAFETY_SAFE);
rdv40_spiffs_make_symlink((char *)HFCOLIN_LASTTAG_SYMLINK,(uint8_t *)dest, RDV40_SPIFFS_SAFETY_SAFE);
return;
}
git diff --shortstat master
25 files changed, 8426 insertions(+), 90 deletions(-)
ah btw I've 'taken' quite some space in the pm3_cmd.h
so just to avoid collision in between these days :
// RDV40, High level flashmem SPIFFS Manipulation
// ALL function will have a lazy or Safe version
// that will be handled as argument of safety level [0..2] respectiveley normal / lazy / safe
// However as how design is, MOUNT and UNMOUNT only need/have lazy as safest level so a safe level will still execute a lazy version
// see spiffs.c for more about the normal/lazy/safety information)
#define CMD_SPIFFS_MOUNT 0x0130
#define CMD_SPIFFS_UNMOUNT 0x0131
//Seeking, APPEND, MODES and FLAGS will be arguments of command (O_WR/RD,O_CREAT,O_TRUNC,O_APPEND etc etc)
#define CMD_SPIFFS_WRITE 0x0132
#define CMD_SPIFFS_READ 0x0133
//We use no open/close instruvtion, as they are handled internally.
#define CMD_SPIFFS_REMOVE 0x0134
#define CMD_SPIFFS_RM CMD_SPIFFS_REMOVE
#define CMD_SPIFFS_RENAME 0x0135
#define CMD_SPIFFS_MV CMD_SPIFFS_RENAME
#define CMD_SPIFFS_COPY 0x0136
#define CMD_SPIFFS_CP CMD_SPIFFS_COPY
#define CMD_SPIFFS_STAT 0x0137
#define CMD_SPIFFS_FSTAT 0x0138
#define CMD_SPIFFS_INFO 0x0139
#define CMD_SPIFFS_FORMAT CMD_FLASHMEM_WIPE
// This take a +0x1000 as they are high level helper and special functions
// As the others, they may have safety level argument if it makkes sense
#define CMD_SPIFFS_PRINT_TREE 0x1130
#define CMD_SPIFFS_GET_TREE 0x1131
#define CMD_SPIFFS_TEST 0x1132
// more ?
this sounds very promising :blush:
the FS extends to which portion of the Flash? First 3/4? Or also encompassing the password dictionaries and t55xx settings? which would make sense...
I used whole flash but juste modifying the config to use only half of flash or 3/4 is possible. I'll modify the default config to be compliant with flash layout and will not touch the upper 128kb. Do well have by default 128kb for spiffs, page 4 for internals, page 3 available for user needs out of spiffs. Logical page size in spiffs has been made 256byte, and physical size are 4kb since it is the minimum we can erase.
This is actual config :
#define SPIFFS_CFG_PHYS_SZ (1024 * 256)
#define SPIFFS_CFG_PHYS_ERASE_SZ (4 * 1024)
#define SPIFFS_CFG_PHYS_ADDR (0)
#define SPIFFS_CFG_LOG_PAGE_SZ (256)
#define SPIFFS_CFG_LOG_BLOCK_SZ (4 * 1024)
#define LOG_PAGE_SIZE 256
so
#define SPIFFS_CFG_PHYS_SZ (1024 * 256)
will be modified to
#define SPIFFS_CFG_PHYS_SZ (1024 * 128)
.
At the lowest level that's what I gave to SPIFFS :
///// FLASH LEVEL R/W/E operations for feeding SPIFFS Driver/////////////////
static s32_t rdv40_spiffs_llread(u32_t addr, u32_t size, u8_t *dst) {
Flash_ReadData(addr, dst, size);
return SPIFFS_OK;
}
static s32_t rdv40_spiffs_llwrite(u32_t addr, u32_t size, u8_t *src) {
if (!FlashInit()) {
return false;
}
Flash_Write(addr, src, size);
return SPIFFS_OK;
}
static s32_t rdv40_spiffs_llerase(u32_t addr, u32_t size) {
/*if (addr & (size*1024 - 1)) {
if ( DBGLEVEL > 1 ) Dbprintf("Flash_Erase4k : Address is not align at 4096");
return false;
}*/
if (!FlashInit()) {
return 128;
}
uint32_t bytes_erased = 0, bytes_remaining = size;
while (bytes_remaining > 0) {
addr += bytes_erased;
Flash_CheckBusy(BUSY_TIMEOUT);
Flash_WriteEnable();
FlashSendByte(SECTORERASE);
Flash_TransferAdresse(addr);
FlashSendLastByte(0);
bytes_remaining -= 4096;
bytes_erased += 4096;
}
Flash_CheckBusy(BUSY_TIMEOUT);
FlashStop();
return SPIFFS_OK;
}
////////////////////////////////////////////////////////////////////////////////
Oh my! this CopyPaste made me spot a terrible mistake :)
A failed FlashInit
should return !false
just as in the llerase
since SPIFFS_OK
is actually a 0
.
We avoided some SPIFFSGate here.
BTW this is normal that a failed read can generate a SPIFFS_OK
: it will generate a bad file descriptor and permit a Handling the error at higher level which is very desirable when working with both read and write caches.
I used whole flash
Just make sure you don't touch the very last sector (4kb) because it contains the flash signature delivered by RRG.
My signature sector was destroyed long time ago. Don't worry about this, I made the new default configuration first half of the spi flash.
ok, new delay...
Was working a little more deep around writing cache to avoid destroying our flash with a simple fibonacci bomb, and then it popped out.
ISO C99 doesn't support unnamed structs/unions
And I ensure, unnamed structs/unions there are. More than I am able to handle today.
I'm sorry showing here my own personal issues but... The spiffs author actually depending on the point of view either
#define
added, anonymous structs in anonymous unions in anonymous structs I won't start a debate it will end right when a random one will remind everyone that anonymous structs and unions were available in Dennis Ritchie's 1974 C compiler, thus far before C89 standard, but still in our quest for some as portable as possible code thus pendanticist C99... well I had a good evening, but this was not this one. :)
Done.
All three read
, write
and temporal fd
caching are working flawlessly and have been rewritten to be C99 compliant. IX_mapping optimizations too. Also provided a second fd and upped the STACK_COPYBUFFER_SIZE
to LOG_PAGE_SIZE
, alongside with 4 logical pages (+FD size + eventual extra metadata) for cache(s).
All optimized, the whole SPIFFS implementation takes a little less than 2Kilobytes of RAM.
Flash size however is top of the range with ~25Kilobytes but since you've been so effective in making our available flash space twice it's former, that should not be a problem.
A little thing that will need tuning however will be the garbage collector, as it is blocking operation so we need to have some real life feedbacks on actual usage of the spiffs to provide more accurate settings, at least for the GC_MAX_RUNS
. Tweaking the heuristics are beyond what should be anyhow needed.
We'll also have to wait for confirmation that we can maintain the PAGE_CHECK
mechanisms disabled as they tend have very strong drawback on performances. There are quite other mecanism which ensure integrity, with the tradeoff that anything still not flushed from cache when let's say a powerloss occurs will be lost.
However since the SAFE
safety level I implemented always rollback to previous state if there was a change in mounted state, and since everything is gracefully closed and flushed upon unmouting operations, if one never manually mount the filesystem beforehand and then always rely on the SAFE safety level (which is more than enough and effective for ponctual read/writes, like my standalone), then cache flush will never be an issue, and a loss of any cache would in fact represent a case where we just avoided a corrupted state in a bunch of pages.
Hum as things are going, I think if someone is motivated enough, we have everything needed to implement a VFS api with POSIX-oriented calls and FILE objects. This would make "userland" filesystem manipulation way easier and standard. I'll add everything underlyingly needed to the Monday PR.
@iceman1001 @doegox @slurdge if anyone has an strtok
either optimized or stripped implementation...
arm-none-eabi-nm -S -r --size-sort obj/string.o
00000000 00000058 T itoa
00000000 00000056 T __strtok_r
00000000 00000030 T strncpy
00000000 0000002e T strncat
00000000 00000026 T strreverse
00000000 00000026 T strcat
00000000 00000024 T memcmp
00000000 0000001e T strcmp
00000000 0000001a T memxor
00000000 00000018 T memset
00000000 00000018 T memcpy
00000000 00000014 T strtok
00000000 00000012 T strcpy
00000000 00000010 T strlen
00000000 00000004 b last.4207
optimizing 106 bytes? is it such a big deal ?
Yeah ive made something correct size wise but I don't think it is that good neither performance wise nor as safety wise. Remember I was the standalone with the worst overflow possible going through commits after commits. I'm not really reliable when it comes to string manipulation.
Aaaah the pleasure of having to modify GetFromDevice
so we can pass Filenames (up to SPIFSS_OBJ_NAME_LEN so 32bytes here) and then having to modify each instance of it's use :)
yipi!
Insight : I absolutely Hate the client part of the proxmark3 :) In fact C is either embedded and no-os or either better replaced with something more High level.
and here we go :
[usb] pm3 --> mem wipe p 0
[+] Flash WIPE ok
[usb] pm3 --> mem wipe p 1
[+] Flash WIPE ok
[usb] pm3 --> mem spiffs tree
#db# /
Thanks to test command, all this is done internally
[usb] pm3 --> mem spiffs test
#db# ---------------------------
#db# Testing SPIFFS functionning
#db# ---------------------------
#db# (all test are made using lazy safetylevel)
#db# * Mounting filesystem (lazy).......
#db# * Printing tree..............
#db# /
#db# * Writing 'I love Proxmark' in a testspiffs.txt
#db# * Printing tree again.......
#db# /
#db# [0001] 15bytes |-- testspiffs.txt
#db# * Making a symlink to testspiffs.txt
#db# * Printing tree again.......
#db# /
#db# [0001] 15bytes |-- testspiffs.txt
#db# [0002] 32bytes |-- linktotestspiffs(.lnk) --> testspiffs.txt
#db# * Rollbacking The mount status IF things have changed
#db# All done
[usb] pm3 --> mem spiffs info
#db# Flash Memory FileSystem Infos (SPIFFS)
#db# -------------------------------------
#db# * Filesystem Logical Block Size.........4096 bytes
#db# * Filesystem Logical Page Size..........256 bytes
#db# --
#db# * Filesystem Max Open Files.............2 file descriptors
#db# * Filesystem Max Path Lenght............32 chars
#db# --
#db# Filesystem Size Used Available Use% Mounted on
#db# spiffs 113201B 1004B 112197B 1% /
[usb] pm3 --> mem spiffs tree
#db# /
#db# [0001] 15bytes |-- testspiffs.txt
#db# [0002] 32bytes |-- linktotestspiffs(.lnk) --> testspiffs.txt
(so beware the SIZE element should still be the original file size, as the 32bytes size from symlink nothing but the Maximum lenght of a filename)
[usb] pm3 --> mem spiffs dump o testspiffs.txt l 15 f testspiffs.txt
[=] downloading 15 bytes from spiffs (flashmem)
#db# Filename received for spiffs dump :: testspiffs.txt
[+] saved 15 bytes to binary file testspiffs.txt
usb] pm3 --> mem spiffs dump o linktotestspiffs l 15 f testspiffsfromsymlink.txt
[=] downloading 15 bytes from spiffs (flashmem)
#db# Filename received for spiffs dump :: linktotestspiffs
[+] saved 15 bytes to binary file testspiffsfromsymlink.txt
[usb] pm3 --> exit
~/lastpr/spiffs/proxmark3/armsrc master* 47s
❯ cat testspiffs.txt
I love Proxmark~
~/lastpr/spiffs/proxmark3/armsrc master* 30s
❯ cat testspiffsfromsymlink.txt
I love Proxmark~
[usb] pm3 --> mem spiffs tree
#db# /
#db# [0001] 15bytes |-- testspiffs.txt
#db# [0002] 32bytes |-- linktotestspiffs(.lnk) --> testspiffs.txt
#db# [0003] 1024bytes |-- hf_colin/mf_a67f8bf5.bin
#db# [0004] 32bytes |-- hf_colin/lasttag(.lnk) --> hf_colin/mf_a67f8bf5.bin
(both as-is and in actual readable form)
[usb] pm3 --> mem spiffs dump o hf_colin/mf_a67f8bf5.bin l 1024 f mf_a67f8bf5.bin e
[=] downloading 1024 bytes from spiffs (flashmem)
#db# Filename received for spiffs dump :: hf_colin/mf_a67f8bf5.bin
[+] saved 1024 bytes to binary file mf_a67f8bf5.bin
[+] saved 64 blocks to text file mf_a67f8bf5.bin.eml
[usb] pm3 --> quit
~/lastpr/spiffs/proxmark3/client master* 48s
❯ head -n16 mf_a67f8bf5.bin.eml
A67F8BF5A7880400468EXXXXXXXXXXXX
01074E00000000000000000000000000
55550000000000000000000000000000
1122334455667F078800112233445566
0000000058D0286A0100000000B27D3F
41151100000000000000000000010000
000000000000000000000000F0000000
1122334455667F078800112233445566
XXXXXXXX000000FFFFFFFFFFFFFFFFFF
XXXXXXXXXXXXXXXXXXXXXXXXXX000000
00000000000000000000000000000000
1122334455667F078800112233445566
00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000
1122334455667F078800112233445566
~/lastpr/spiffs/proxmark3/client master*
❯ hexdump -C mf_a67f8bf5.bin | head -n16
00000000 a6 7f 8b f5 a7 88 04 00 46 8e xx xx xx xx xx xx |........F.xxxxxx|
00000010 01 07 4e 00 00 00 00 00 00 00 00 00 00 00 00 00 |..N.............|
00000020 55 55 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |UU..............|
00000030 11 22 33 44 55 66 7f 07 88 00 11 22 33 44 55 66 |...........)....|
00000040 00 00 00 00 58 d0 28 6a 01 00 00 00 00 b2 7d 3f |....X.(j......}?|
00000050 41 15 11 00 00 00 00 00 00 00 00 00 00 01 00 00 |A...............|
00000060 00 00 00 00 00 00 00 00 00 00 00 00 f0 00 00 00 |................|
00000070 11 22 33 44 55 66 7f 07 88 00 11 22 33 44 55 66 |...........)....|
00000080 xx xx xx xx 00 00 00 ff ff ff ff ff ff ff ff ff |xxxxxxxxxxxxxxxx|
00000090 xx xx xx xx xx xx xx xx xx xx xx xx xx 00 00 00 |xxxxxxxxxxxx....|
000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000b0 11 22 33 44 55 66 7f 07 88 00 11 22 33 44 55 66 |...........)....|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000000f0 11 22 33 44 55 66 7f 07 88 00 11 22 33 44 55 66 |...........)....|
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
Okay we are in the very last moment before Cleaning/merging/sending PR. dump AND load operation works flawlessly without need of known size, since it's either handled by first sending STAT command on file preexistance or transparently handled on device-side. Still needs some sanity check both in client and device but after all that work without real pause I just can't manage to check for all this again, nor I'm motivated to. So yes, we can destroy some buffers and a lots of checks should be updated accordingly on both sides.
Anyway :
[usb] pm3 --> mem wipe p 0
[+] Flash WIPE ok
[usb] pm3 --> mem wipe p 1
[+] Flash WIPE ok
[usb] pm3 --> mem spiffs tree
#db# /
(without other treatment directly in the device filesystem) (Note that size is not needed argument now being handled client side and sent along)
[usb] pm3 --> mem spiffs load f ../client/default_keys.dic o default_keys.dic
[+] loaded 15204 bytes from binary file ../client/default_keys.dic
[+] Wrote 15204 bytes to file 3902861296
[usb] pm3 --> mem spiffs tree
#db# /
#db# [0001] 15204bytes |-- default_keys.dic
(again no size, handled by STAT command sent during the process)
[usb] pm3 --> mem spiffs dump o default_keys.dic f dumped_def_keys.dic
[=] downloading 15204 bytes from spiffs (flashmem)
[+] saved 15204 bytes to binary file dumped_def_keys.dic
[usb] pm3 --> exit
~/lastpr/spiffs/proxmark3/armsrc master* 47s
❯ md5 dumped_def_keys.dic
MD5 (dumped_def_keys.dic) = 3b6d77ce0e6acbf18f556e00ff206787
~/lastpr/spiffs/proxmark3/armsrc master*
❯ md5 ../client/default_keys.dic
MD5 (../client/default_keys.dic) = 3b6d77ce0e6acbf18f556e00ff206787
Yihay.
Got a LOT of mess to clean and other things to being able to merge, but I'm actually DONE for everything else.
Boom ! let's get hands dirty and track the mess ! May also have that much of relicates or dirtyness, help me clean the mess people of RDV40 !
Now this is a PR ! Enormous work done!
A suggestion, shouldn't all return codes be defines, instead of mixing "SPIFF_OK" vs false.
@doegox started the work with return codes on clientside. PM3_SUCCESS vs PM3_ESOFT etc.
We could reuse those for device side aswell. So we don't have to keep track of all different defines when coding.
For your test you can call direct from shell now :) proxmark3 -c 'mem dump' | md5
Thanks!
Yes they should as long as when they should not (e.g. return 0 == SPIFFS_OK
).
Any standardisation / homogenization is good to take.
About the dump mem dump
and mem spiffs dump
have bunch of uncommon so did not want to test it with black pipe magics added etc until it's more solid.
aha, it wasn't about mem dump specific, it was just to mention you don't have to enter the client and exit while testing. Just call the client with -c and your commands you want to run, it will execute and exit ... :)
Oh OK. I will remember :D anyway I'm closing this since it was merged a little sooner :) I @iceman1001 you may keep it as some base for documentation / example? I don't know
we will have to make documentation under /doc/notes_on_spiffs.md
better to merge it in the existing doc/ext_flash_notes.md imho could you also adapt hf_bog to use spiffs? else it will savagely write over your spiffs... maybe we should have another issue to discuss these little things around spiffs ?
of course we could.
hf bog yes, thats next issue. and the configure etc.
Issue opened for such matter.
Is your feature request related to a problem? Please describe. No. Unless we consider the low level way we manage flash a problem Describe the solution you'd like Implementation of Spiffs on customizable part of flash (64kb 128kb or 256kb) Describe alternatives you've considered Carrot cake. Did not yield to any comparable results'
Additional context This is in fact my own request since I've managed to implement it with 256bytes logic pages and 4k blocks, write read and temporal cachrme with all three sizes. I'm currently opening this for further ideas and discussion on this idea and how things can go since this open a lot of new possibilities. For the record, the implementation takes an overall 1kb ram with caches since we don't need more than one fd at a time. I think a full feature 4fd implementation will not be more than 2kb ram. Binary size is another beast though, since this singleton instance will not be less than a 20kb size in flash. However a read-only version if we decide to let the write function a client thing is then less than 9kb
Let's discuss to make a PR not subject to immediate 50% invalid :)