如何压缩url上json参数

问题描述

由于一些业务场景，跳转到对应的页面需要在其页面的URL上带上大量的json（包含中文字符），但是这个页面由于链接长度过长会导致页面中的请求被截断，发生一些不可预知的情况，所以如何优化业务的这种情况？

思路

1. pack算法压缩

1.1 实现代码：

现有的npm package jsonpack实现了json的压缩算法，可直接调用使用：

// big JSON
const json = {...}

// pack the big JSON 
const packed = jsonpack.pack(json);

// do stuff...

// And then unpack the packed
const json = jsonpack.unpack(packed);

1.2 压缩结果

通过pack算法压缩，压缩之后关键信息长度：531，对应结果如下：

distance|10.8km|id|3887|store|福溢家居(红星美凯龙至尊Mall)|3890|14.8km|6770|大自然/nature（居然之家玉泉营店）|8.9km|3816|HTL(红星美凯龙北四环商场)|9495|爱依瑞斯(居然之家玉泉营店)|16.5km|4129|HTL(北京西四环商场)|17.3km|8109|MD(居然之家玉泉营店)|8104|11.2km|4370|巴里巴特（红星美凯龙北京家居1号店）|22.0km|1524|顾家（居然之家八角店）|1374|非同(居然之家八角店)|7.0km|302|安东尼奥.观邸（居然之家北四环店）|6206|9166|斯帕尔(居然之家玉泉营店)|4126^^^@$0|1|2|3|4|5]|$0|1|2|6|4|5]|$0|7|2|8|4|9]|$0|A|2|B|4|C]|$0|7|2|D|4|E]|$0|F|2|G|4|H]|$0|I|2|J|4|K]|$0|I|2|L|4|K]|$0|M|2|N|4|O]|$0|P|2|Q|4|R]|$0|P|2|S|4|T]|$0|U|2|V|4|W]|$0|F|2|X|4|H]|$0|7|2|Y|4|Z]|$0|F|2|10|4|H]]

1.3 编码结果

经过encodeURIComponent编码之后，关键信息长度 1853，对应结果如下：

distance%7C10.8km%7Cid%7C3887%7Cstore%7C%E7%A6%8F%E6%BA%A2%E5%AE%B6%E5%B1%85(%E7%BA%A2%E6%98%9F%E7%BE%8E%E5%87%AF%E9%BE%99%E8%87%B3%E5%B0%8AMall)%7C3890%7C14.8km%7C6770%7C%E5%A4%A7%E8%87%AA%E7%84%B6%2Fnature%EF%BC%88%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E7%8E%89%E6%B3%89%E8%90%A5%E5%BA%97%EF%BC%89%7C8.9km%7C3816%7CHTL(%E7%BA%A2%E6%98%9F%E7%BE%8E%E5%87%AF%E9%BE%99%E5%8C%97%E5%9B%9B%E7%8E%AF%E5%95%86%E5%9C%BA)%7C9495%7C%E7%88%B1%E4%BE%9D%E7%91%9E%E6%96%AF(%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E7%8E%89%E6%B3%89%E8%90%A5%E5%BA%97)%7C16.5km%7C4129%7CHTL(%E5%8C%97%E4%BA%AC%E8%A5%BF%E5%9B%9B%E7%8E%AF%E5%95%86%E5%9C%BA)%7C17.3km%7C8109%7CMD(%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E7%8E%89%E6%B3%89%E8%90%A5%E5%BA%97)%7C8104%7C11.2km%7C4370%7C%E5%B7%B4%E9%87%8C%E5%B7%B4%E7%89%B9%EF%BC%88%E7%BA%A2%E6%98%9F%E7%BE%8E%E5%87%AF%E9%BE%99%E5%8C%97%E4%BA%AC%E5%AE%B6%E5%B1%851%E5%8F%B7%E5%BA%97%EF%BC%89%7C22.0km%7C1524%7C%E9%A1%BE%E5%AE%B6%EF%BC%88%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E5%85%AB%E8%A7%92%E5%BA%97%EF%BC%89%7C1374%7C%E9%9D%9E%E5%90%8C(%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E5%85%AB%E8%A7%92%E5%BA%97)%7C7.0km%7C302%7C%E5%AE%89%E4%B8%9C%E5%B0%BC%E5%A5%A5.%E8%A7%82%E9%82%B8%EF%BC%88%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E5%8C%97%E5%9B%9B%E7%8E%AF%E5%BA%97%EF%BC%89%7C6206%7C9166%7C%E6%96%AF%E5%B8%95%E5%B0%94(%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E7%8E%89%E6%B3%89%E8%90%A5%E5%BA%97)%7C4126%5E%5E%5E%40%240%7C1%7C2%7C3%7C4%7C5%5D%7C%240%7C1%7C2%7C6%7C4%7C5%5D%7C%240%7C7%7C2%7C8%7C4%7C9%5D%7C%240%7CA%7C2%7CB%7C4%7CC%5D%7C%240%7C7%7C2%7CD%7C4%7CE%5D%7C%240%7CF%7C2%7CG%7C4%7CH%5D%7C%240%7CI%7C2%7CJ%7C4%7CK%5D%7C%240%7CI%7C2%7CL%7C4%7CK%5D%7C%240%7CM%7C2%7CN%7C4%7CO%5D%7C%240%7CP%7C2%7CQ%7C4%7CR%5D%7C%240%7CP%7C2%7CS%7C4%7CT%5D%7C%240%7CU%7C2%7CV%7C4%7CW%5D%7C%240%7CF%7C2%7CX%7C4%7CH%5D%7C%240%7C7%7C2%7CY%7C4%7CZ%5D%7C%240%7CF%7C2%7C10%7C4%7CH%5D%5D

1.4 压缩效果

压缩比：1853/2750 = 0.673818182

2. 指定分割字符

2.1 实现代码：

使用_作为对象属性分割，使用*作为对象分割，核心实现代码

export function compressPanoListParams (value: PanoramaList, result: string = '') {
  if (value.length === 0) return result;
  if (result) result += '*';
  const { distance, id, store } = value.shift();
  const currentResult = `${distance}_${id}_${store}`;
  return compressPanoListParams(value, result + currentResult);
}

2.2 压缩结果

使用_作为对象属性分割，使用*作为对象分割，压缩之后关键信息长度：404，压缩结果如下：

110.8km_3887_福溢家居(红星美凯龙至尊Mall)10.8km_3890_福溢家居(红星美凯龙至尊Mall)14.8km_6770_大自然/nature（居然之家玉泉营店）8.9km_3816_HTL(红星美凯龙北四环商场)14.8km_9495_爱依瑞斯(居然之家玉泉营店)16.5km_4129_HTL(北京西四环商场)17.3km_8109_MD(居然之家玉泉营店)17.3km_8104_MD(居然之家玉泉营店)11.2km_4370_巴里巴特（红星美凯龙北京家居1号店）22.0km_1524_顾家（居然之家八角店）22.0km_1374_非同(居然之家八角店)7.0km_302_安东尼奥.观邸（居然之家北四环店）16.5km_6206_HTL(北京西四环商场)14.8km_9166_斯帕尔(居然之家玉泉营店)16.5km_4126_HTL(北京西四环商场)

2.3 编码结果

经过encodeURIComponent编码之后，关键信息长度：1670

10.8km3887%E7%A6%8F%E6%BA%A2%E5%AE%B6%E5%B1%85(%E7%BA%A2%E6%98%9F%E7%BE%8E%E5%87%AF%E9%BE%99%E8%87%B3%E5%B0%8AMall)10.8km3890%E7%A6%8F%E6%BA%A2%E5%AE%B6%E5%B1%85(%E7%BA%A2%E6%98%9F%E7%BE%8E%E5%87%AF%E9%BE%99%E8%87%B3%E5%B0%8AMall)14.8km6770%E5%A4%A7%E8%87%AA%E7%84%B6%2Fnature%EF%BC%88%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E7%8E%89%E6%B3%89%E8%90%A5%E5%BA%97%EF%BC%898.9km_3816_HTL(%E7%BA%A2%E6%98%9F%E7%BE%8E%E5%87%AF%E9%BE%99%E5%8C%97%E5%9B%9B%E7%8E%AF%E5%95%86%E5%9C%BA)14.8km9495%E7%88%B1%E4%BE%9D%E7%91%9E%E6%96%AF(%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E7%8E%89%E6%B3%89%E8%90%A5%E5%BA%97)16.5km_4129_HTL(%E5%8C%97%E4%BA%AC%E8%A5%BF%E5%9B%9B%E7%8E%AF%E5%95%86%E5%9C%BA)17.3km_8109_MD(%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E7%8E%89%E6%B3%89%E8%90%A5%E5%BA%97)17.3km_8104_MD(%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E7%8E%89%E6%B3%89%E8%90%A5%E5%BA%97)11.2km4370%E5%B7%B4%E9%87%8C%E5%B7%B4%E7%89%B9%EF%BC%88%E7%BA%A2%E6%98%9F%E7%BE%8E%E5%87%AF%E9%BE%99%E5%8C%97%E4%BA%AC%E5%AE%B6%E5%B1%851%E5%8F%B7%E5%BA%97%EF%BC%8922.0km1524%E9%A1%BE%E5%AE%B6%EF%BC%88%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E5%85%AB%E8%A7%92%E5%BA%97%EF%BC%8922.0km1374%E9%9D%9E%E5%90%8C(%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E5%85%AB%E8%A7%92%E5%BA%97)7.0km302%E5%AE%89%E4%B8%9C%E5%B0%BC%E5%A5%A5.%E8%A7%82%E9%82%B8%EF%BC%88%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E5%8C%97%E5%9B%9B%E7%8E%AF%E5%BA%97%EF%BC%8916.5km_6206_HTL(%E5%8C%97%E4%BA%AC%E8%A5%BF%E5%9B%9B%E7%8E%AF%E5%95%86%E5%9C%BA)14.8km9166%E6%96%AF%E5%B8%95%E5%B0%94(%E5%B1%85%E7%84%B6%E4%B9%8B%E5%AE%B6%E7%8E%89%E6%B3%89%E8%90%A5%E5%BA%97)16.5km_4126_HTL(%E5%8C%97%E4%BA%AC%E8%A5%BF%E5%9B%9B%E7%8E%AF%E5%95%86%E5%9C%BA)

2.4 压缩效果

压缩比：1670 / 2750 = 0.607272727

3. Gzipped 压缩

3.1 实现代码

使用Node中的zlib进行压缩处理，核心实现代码如下：

export function compressPanoListParams (value: PanoramaList, result: string = '') {
  if (value.length === 0) return result;
  if (result) result += '*';
  const { distance, id, store } = value.shift();
  const currentResult = `${distance}_${id}_${store}`;
  return compressPanoListParams(value, result + currentResult);
}

3.2 压缩结果

通过gzip 压缩对应json压缩之后，关键信息长度：544，结果如下：

export function gzipPanoListParams(panoramaList: PanoramaList): string {
  return zlib.gzipSync(JSON.stringify(panoramaList)).toString('base64');
}

H4sIAAAAAAAEAK2SS0sCURiG/8usDGSam3Np3aJF7tpFC0kXkhmoraKNZOOUqEVl4SWSJqHCC4aXQaY/M99cVv6FTgR5ZjSzy+ZwDofznOflezcPiHA0mQrFtyPECkFTpLizS/iJaBidWFEU0D6Z2ku8X1oPBVOrQ7MHnYzP0urm9a2l50FuOfqNLXehfRIMxWJLxKF/LlOi/oHJ4UxeEHAm3Dds+dE66i3HQ6n9RGQ8yiI6OhvDU/STlVfMrmIXVdBK45HiRYukhNvSPEZe21j3SkKuBOWylW/B5TFUtG9MJU4K4OmzHUOvWmc186rl+8pxGsmTgQmSoxnJo4icDO3ZVl/nmwkkO8GINIVjgqs/8PGCuN+CaJLBgrHuqfZfHDmHVksZoolOjwFF/ugRDYX+7NkyDElNPqADDG7q3OnovacskHmyG+cL0VjBRavWoJjzzWRNBRdwEEsxeOymYgwq0B6BqpJ2I+2kB17FzwbOtHS3hWcob6EXbIu7xzSPY1B9YXAJ7Ys/lHhRra03HuUFpbIEAAA=

3.3 编码结果

经过encodeURIComponent编码之后，关键信息长度：582，结果如下：

H4sIAAAAAAAEAK2SS0sCURiG%2F8usDGSam3Np3aJF7tpFC0kXkhmoraKNZOOUqEVl4SWSJqHCC4aXQaY%2FM99cVv6FTgR5ZjSzy%2BZwDofznOflezcPiHA0mQrFtyPECkFTpLizS%2FiJaBidWFEU0D6Z2ku8X1oPBVOrQ7MHnYzP0urm9a2l50FuOfqNLXehfRIMxWJLxKF%2FLlOi%2FoHJ4UxeEHAm3Dds%2BdE66i3HQ6n9RGQ8yiI6OhvDU%2FSTlVfMrmIXVdBK45HiRYukhNvSPEZe21j3SkKuBOWylW%2FB5TFUtG9MJU4K4OmzHUOvWmc186rl%2B8pxGsmTgQmSoxnJo4icDO3ZVl%2FnmwkkO8GINIVjgqs%2F8PGCuN%2BCaJLBgrHuqfZfHDmHVksZoolOjwFF%2FugRDYX%2B7NkyDElNPqADDG7q3OnovacskHmyG%2BcL0VjBRavWoJjzzWRNBRdwEEsxeOymYgwq0B6BqpJ2I%2B2kB17FzwbOtHS3hWcob6EXbIu7xzSPY1B9YXAJ7Ys%2FlHhRra03HuUFpbIEAAA%3D

3.4 压缩效果

压缩比：582 / 2750 = 0.211636364

4. 方案总结

方案	压缩比	复杂度
pack算法压缩	0.673818182	中等
指定分割字符	0.607272727	简单
Gzipped 压缩	0.211636364	复杂

最终方案

1. 来源页服务端

1.1 分割

使用_作为对象属性分割，使用*作为对象分割，压缩有长度：（关键信息长度：404）

export function compressPanoListParams (value: PanoramaList, result: string = '') {
  if (value.length === 0) return result;
  if (result) result += '*';
  const { distance, id, store } = value.shift();
  const currentResult = `${distance}_${id}_${store}`;
  return compressPanoListParams(value, result + currentResult);
}

110.8km_3887_福溢家居(红星美凯龙至尊Mall)10.8km_3890_福溢家居(红星美凯龙至尊Mall)14.8km_6770_大自然/nature（居然之家玉泉营店）8.9km_3816_HTL(红星美凯龙北四环商场)14.8km_9495_爱依瑞斯(居然之家玉泉营店)16.5km_4129_HTL(北京西四环商场)17.3km_8109_MD(居然之家玉泉营店)17.3km_8104_MD(居然之家玉泉营店)11.2km_4370_巴里巴特（红星美凯龙北京家居1号店）22.0km_1524_顾家（居然之家八角店）22.0km_1374_非同(居然之家八角店)7.0km_302_安东尼奥.观邸（居然之家北四环店）16.5km_6206_HTL(北京西四环商场)14.8km_9166_斯帕尔(居然之家玉泉营店)16.5km_4126_HTL(北京西四环商场)

1.2. 压缩

指定字符进行压缩 + gzip压缩（字符编码为base64格式） (关键信息长度：480)

export function gzipPanoListParams(panoramaList: PanoramaList): string {
  return zlib.gzipSync(JSON.stringify(panoramaList)).toString('base64');
}

H4sIAAAAAAAEAI2Ry07CQBiFXwdYjDNtmbZ7Fy5k537CwpXowugLELFUCaAR0XAxEiuJGi7BcGlIfZn+nXbFKzgU0kVVZDNJM+ecb3oOIRhpR8dM1jSV8deKZ3egN4ZhIcHtjvfwxJ0yGP3AefSNEQyuMtlcLpmKPDre3qOEHqqqmMFL1zfe+MV45yR7dn56uJgXhVt8u7NrkcTLpjcy/aoFdn0xN1Ma0kMaoWzvYD8OgVIdGg1e7kPtEpp2RNIVPc14ceg6LX7T9u77ib8YwkJRWlgUIukhQmS69odvfcWSVSQLmUawzjK7m/IiofKPkCBpCZaXrUw+A6MkTm7ORCM/f1M8adUzgcpk3Y0kISwCSFpSWPDsiPtYmVB497u3MbWsCnWrDdVS4ldtMqWGQhlLDHqmO23CYA6WhfxuPshP44hogRVl3SaVMN3Y5nonQikT88C0BoO7rUbaFPsNSNCimdECAAA=

1.3. 编码

使用encode针对参数进行转码处理（关键信息长度：496）

H4sIAAAAAAAEAI2Ry07CQBiFXwdYjDNtmbZ7Fy5k537CwpXowugLELFUCaAR0XAxEiuJGi7BcGlIfZn%2BnXbFKzgU0kVVZDNJM%2Becb3oOIRhpR8dM1jSV8deKZ3egN4ZhIcHtjvfwxJ0yGP3AefSNEQyuMtlcLpmKPDre3qOEHqqqmMFL1zfe%2BMV45yR7dn56uJgXhVt8u7NrkcTLpjcy%2FaoFdn0xN1Ma0kMaoWzvYD8OgVIdGg1e7kPtEpp2RNIVPc14ceg6LX7T9u77ib8YwkJRWlgUIukhQmS69odvfcWSVSQLmUawzjK7m%2FIiofKPkCBpCZaXrUw%2BA6MkTm7ORCM%2Ff1M8adUzgcpk3Y0kISwCSFpSWPDsiPtYmVB497u3MbWsCnWrDdVS4ldtMqWGQhlLDHqmO23CYA6WhfxuPshP44hogRVl3SaVMN3Y5nonQikT88C0BoO7rUbaFPsNSNCimdECAAA%3D

1.4. 效果

496 / 2750 = 0.180363636

2. 承接页服务端

2.1. 解码

function getPageParams(pageUrl: string): PageParams {
  if (!pageUrl) return {};
  const decodePageUrl = decodeURIComponent(pageUrl);
  const pageQuery = url.parse(decodePageUrl).query || "";
  /**
   * @remarks 为什么需要replace
   * 按照RFC-3986规范，空格被编码成%20，而加号"+"被编码成%2B
   * 如果不进行replace，对应的+会变为空格
   */
  return qs.parse(pageQuery.replace(/\+/g, '%2B')) as unknown as PageParams;
}

2.2. 解压缩

export function unzipPanoListParams(value: string): PanoramaList {
  try {
    const panoramaJson = zlib
      .unzipSync(Buffer.from(value, "base64"))
      .toString();
    return formattingParams(panoramaJson);
  } catch (error) {
    return [];
  }
}

2.3. 解分割

export function inflatePanoListParams(params: string): PanoramaList {
  const result: PanoramaList = [];
  const objects = params.split("*");
  for (const objectItem of objects) {
    const objectAttribute = objectItem.split("_");
    result.push({
      distance: objectAttribute[0],
      id: objectAttribute[1],
      store: objectAttribute[2],
    });
  }
  return result;
}

问题记录

1. url解析

针对带有base64的url参数，使用qs.parse进行解析，会将base64中的+变为空格，所以需要使用replace(/\+/g, '%2B'))提前进行处理

  private static getPageParams(pageUrl: string): PageParams {
    if (!pageUrl) return {};
    const decodePageUrl = decodeURIComponent(pageUrl);
    const pageQuery = url.parse(decodePageUrl).query || "";
    /**
     * @remarks 为什么需要replace
     * 按照RFC-3986规范，空格被编码成%20，而加号"+"被编码成%2B
     * 如果不进行replace，对应的+会变为空格
     */
    return qs.parse(pageQuery.replace(/\+/g, '%2B')) as unknown as PageParams;
  }

Genluo / Precipitation-and-Summary