Pre-scrub mod/plugin JSON info files

[itzg]

I have found at least one mod, World Overhaul, where the description string contained control character newlines, such as the content below. That is not proper JSON and results in a Jackson deserialization exception (further down). It appears the JSON content needs to be pre-scrubbed converting Ctrl-R and Ctrl-N to \\n.

[
{
  "modid": "worldoverhaulmod",
  "name": "World Overhaul Mod",
  "description": "The idea of this mod is to add items which will add the possibility to change your gamestyle according to your favorite class.
It contains decorative items, new resources and tools. This mod also has a decent amount of items themed by different classes which can be turned of in the config file if they doesn't fit your server or world.

In future updates I will add more items themed by other game roles.",
  "version": "1.861",
  "mcversion": "1.8",
  "url": "",
  "updateUrl": "",
  "authorList": ["Anakin"],
  "credits": "",
  "logoFile": "",
  "screenshots": [],
  "dependencies": []
}
]

and the observed exception

16:40:37.053 [main] WARN  m.i.mccy.services.ZipMiningService - Handler me.itzg.mccy.services.ModsService$$Lambda$7/837246227@47874b25 failed to process zip content file named {}
com.fasterxml.jackson.databind.JsonMappingException: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value
 at [Source: org.springframework.util.StreamUtils$NonClosingInputStream@290b1b2e; line: 5, column: 146] (through reference chain: java.util.ArrayList[0]->me.itzg.mccy.model.FmlModListEntry["description"])
    at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210)
    at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:177)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1474)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:341)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2810)
    at me.itzg.mccy.services.ModsService.extractFmlModInfo(ModsService.java:256)

[itzg]

Saw a different type of issue in this area with the geochest mod

08:18:09.314 [main] WARN  m.i.mccy.services.ZipMiningService - Handler me.itzg.mccy.services.ModsService$$Lambda$7/837246227@60cf80e7 failed to process zip content file named {}
com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 start byte 0xa7
 at [Source: org.springframework.util.StreamUtils$NonClosingInputStream@24f360b2; line: 4, column: 11] (through reference chain: java.util.ArrayList[0]->me.itzg.mccy.model.FmlModListEntry["name"])
    at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210)
    at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:177)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1474)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:341)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2810)
    at me.itzg.mccy.services.ModsService.extractFmlModInfo(ModsService.java:253)
    at me.itzg.mccy.services.ModsService.processFmlModInfo(ModsService.java:193)
    at me.itzg.mccy.services.ModsService.lambda$traverseZip$1(ModsService.java:165)
    at me.itzg.mccy.services.impl.ZipMiningServiceImpl.lambda$interrogate$1(ZipMiningServiceImpl.java:52)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
    at me.itzg.mccy.services.impl.ZipMiningServiceImpl.interrogate(ZipMiningServiceImpl.java:49)
    at me.itzg.mccy.services.ModsService.traverseZip(ModsService.java:159)
    at me.itzg.mccy.services.ModsService.ingest(ModsService.java:113)
    at me.itzg.mccy.services.ModsServiceTest.lambda$testBulkSuite$1(ModsServiceTest.java:190)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
    at me.itzg.mccy.services.ModsServiceTest.testBulkSuite(ModsServiceTest.java:184)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xa7
 at [Source: org.springframework.util.StreamUtils$NonClosingInputStream@24f360b2; line: 4, column: 11]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581)
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3467)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3461)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2488)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2414)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:285)
    at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:32)

[itzg]

There was also a case of unquoted version strings:

[
    {
        "modid": "parachutemod",
        "name": "Parachute Mod",
        "description": "Adds a Parachute to the game. Jump off cliffs! Base Jumping!! Soar to new heights!",
        "version": 1.7.10-2.5.6,
        "mcversion": 1.7.10,
        "url": "",
        "updateUrl": "",
        "authorList": [ "crackedEgg" ],
        "credits": "Created by crackedEgg",
        "logoFile": "pack.png",
        "screenshots": [],
        "parent": "",
        "dependencies": [ "mod_MinecraftForge" ]
    }
]

[sshipway]

... any data format coming from somewhere else has to be treated as completely unreliable :)

[itzg]

Very true :) This fix is now deployed on the staging cluster.

[sshipway]

Is there a URL I can connect/test with?

[itzg]

Yep, https://mccy-staging.itzg.me

[sshipway]

Great -- I have a feeling you told me that before. I'll run some tests in between tasks here and let you know of anything I manage to break...

[sshipway]

Plugins with no mcversion defined (eg, dynmap) break the upload

[sshipway]

Some big Bukkit plugins, such as mUltiverse and PermissionsEx also fail due to no mcversion option. I suggest that, if mcversion is not set, assume plugin works for ALL versions

[itzg]

Good find. I'll also check on the mod/plugin suggestion logic to cover that angle of it too.