Yes, I'm going to add a couple of options in the next update:
yes/no named color in the block
all/none/some block color (in some, you can, for example, say red and green appear on the block)
various named color palettes (currently there are ~140 of them) with smaller number of colors (say you want all rgb colors get mapped to only red/yellow/green/blue/purple)
I'm using this to make my notes still readable:
.rm-block-ref.has-annotation span { background-color: unset !important; }