Properly handle fractional layer minzoom and maxzoom

[ahocevar]

This should have been fixed with #1032, but it did not. The resulting resolutions were still NaN.