search

mozq / dencode-web

Source code of DenCode.
https://dencode.com
Apache License 2.0
69 stars 14 forks source link

Fair use of the API? #3

Closed hlaueriksson closed 1 month ago

hlaueriksson commented 1 year ago

Hi!

This is a great project! ❤️

I would like to write and publish a PowerToys Run plugin that use your API.

Can I POST to https://dencode.com/dencode from the plugin? Would that be fair use of the API? Would you prefer that I set a specific user agent for those requests?

Thanks!

/ Henrik

mozq commented 1 year ago

Hi Henrik,

Thank you for asking. I am glad that you are using DenCode.

Unfortunately, I would like you to refrain from calling the API endpoint from other system. The README describes how to deploy to Google Cloud, so could you build your own server?

If you only want to use the endpoint, you can disable the UI by comment out like this:

/src/main/java/com/dencode/web/servlet/pages/IndexServlet.java

//@WebServlet("")
public class IndexServlet extends AbstractDencodeHttpServlet {

If you have any other questions, feel free to ask.

Thank you!

hlaueriksson commented 1 year ago

Thanks for your reply.

Do you know of any public API that does what DenCode do?

mozq commented 1 year ago

I made DenCode because it doesn't exist in the world ;)

/dencode endpoint is designed specifically for the web UI, so it is not general purpose and its specifications are not stable. If you don't mind, try using /dencode endpoint on a trial basis. Please add a header like this: "X-Application-Id: com.laueriksson.henrik.powertoysrun-xxxx-plugin", if you call it. (Please name the ID appropriately and let me know later.) I may ask you to stop if the server load increases, but let's try!

You can use the following icon for the search result. https://github.com/mozq/dencode-web/tree/master/res/favicon

Maybe the plugin that copies a result to the clipboard? I don't know what kind of specifications the new plug-in has, but it would be great if you could also display the link to DenCode site as shown below.

[method-name] value

e.g.

test-value

-> link to https://dencode.com/?v=test-value

string test-value

-> link to https://dencode.com/string?v=test-value

string/base64 test-value
base64 test-value

-> link to https://dencode.com/string/base64?v=test-value

hex test-value

-> link to https://dencode.com/string/hex?v=test-value -> link to https://dencode.com/number/hex?v=test-value

hlaueriksson commented 1 year ago

Yes, exactly! I was not able to find any other service that is as good as DenCode.

I need a couple of weeks to finish other stuff, before I can start on a encode/decode plugin for PowerToys Run. I will try out your API and be nice about it. But I have a feeling that depending on libs, rater than an API, would be better for the plugin I have in mind. I'll keep you posted in this issue.

mozq commented 1 year ago

Of course, I'm in favor of using local libraries. It will not be affected by network issues, and the performance will be good.

The main reason DenCode is encoding on the server is because JavaScript is not good at character encoding conversion. There are no restrictions in C#.

I'm a Mac user but will contribute if possible. I look forward to the plugin being released!

hlaueriksson commented 1 year ago

Hi @mozq

Now I got time to start trying out your API.

Can I use this header: X-Application-Id: Community.PowerToys.Run.Plugin.DenCode

mozq commented 1 year ago

Hi @hlaueriksson Sure. Please let me know if you have any questions.

hlaueriksson commented 1 year ago

I have created a proof of concept of a DenCode PowerToys Run plugin: Community PowerToys Run Plugin DenCode

Have you noticed my requests to the API?

mozq commented 1 year ago

Did you set the cookie header and get an error? I think you have already dealt with it. Cookie header are unnecessary and will be removed. No other major errors were logged, so please go ahead.

mozq commented 1 year ago

I can see your screen record now. Great!!

If the input value and the result value are the same, it may be easier to find the desired result if you put them later in the display order or hide them. Anyway, it seems to be easy to use, and smart behavior.

hlaueriksson commented 1 year ago

Did you set the cookie header and get an error? I think you have already dealt with it. Cookie header are unnecessary and will be removed.

No, I did not set any cookies.

If the input value and the result value are the same, it may be easier to find the desired result if you put them later in the display order or hide them.

Yes, that's smart. I will try this approach.

hlaueriksson commented 1 year ago

I'm parsing the config.properties and messages_en.properties files, to generate data for the plugin to use.

I found two issues:

hlaueriksson commented 1 year ago

I have a question.

When the web is sending a POST request with method all.all, the payload looks like this:

{
    "type": "all",
    "method": "all.all",
    "value": "Hello, world!",
    "oe": "UTF-8",
    "nl": "crlf",
    "tz": "Africa/Algiers",
    "options": {
        "decStrMorseCodeVariant": "international",
        "decCipherCaesarShift": "-3",
        "decCipherEnigmaMachine": "I",
        "decCipherEnigmaReflector": "UKW-A",
        "decCipherEnigmaRotor4": "Beta",
        "decCipherEnigmaRotor3": "I",
        "decCipherEnigmaRotor2": "II",
        "decCipherEnigmaRotor1": "III",
        "decCipherEnigmaReflectorRing": "1",
        "decCipherEnigmaRotor4Ring": "1",
        "decCipherEnigmaRotor3Ring": "1",
        "decCipherEnigmaRotor2Ring": "1",
        "decCipherEnigmaRotor1Ring": "1",
        "decCipherEnigmaReflectorPosition": "1",
        "decCipherEnigmaRotor4Position": "1",
        "decCipherEnigmaRotor3Position": "1",
        "decCipherEnigmaRotor2Position": "1",
        "decCipherEnigmaRotor1Position": "1",
        "decCipherEnigmaPlugboard": "",
        "decCipherEnigmaUhr": "0",
        "decCipherEnigmaUkwd": "",
        "decCipherScytaleKey": "2",
        "decCipherRailFenceKey": "2",
        "encStrBinSeparatorEach": "",
        "encStrHexSeparatorEach": "",
        "encStrHexCase": "lower",
        "encStrURLEncodingSpace": "",
        "encStrBase64LineBreakEach": "",
        "encStrAscii85Variant": "z85",
        "encStrUnicodeEscapeSurrogatePairFormat": "",
        "encStrProgramStringQuotes": "double",
        "encStrMorseCodeVariant": "international",
        "encStrLineSortOrder": "asc",
        "encNumEnShortScaleFractionalPartNotation": "",
        "encNumEnShortScaleSystem": "",
        "encDateISO8601DecimalMark": ".",
        "encDateISO8601ExtDecimalMark": ".",
        "encDateISO8601WeekDecimalMark": ".",
        "encDateISO8601OrdinalDecimalMark": ".",
        "encColorRGBFnNotation": "percentage",
        "encCipherCaesarShift": "-3",
        "encCipherEnigmaMachine": "I",
        "encCipherEnigmaReflector": "UKW-A",
        "encCipherEnigmaRotor4": "Beta",
        "encCipherEnigmaRotor3": "I",
        "encCipherEnigmaRotor2": "II",
        "encCipherEnigmaRotor1": "III",
        "encCipherEnigmaReflectorRing": "1",
        "encCipherEnigmaRotor4Ring": "1",
        "encCipherEnigmaRotor3Ring": "1",
        "encCipherEnigmaRotor2Ring": "1",
        "encCipherEnigmaRotor1Ring": "1",
        "encCipherEnigmaReflectorPosition": "1",
        "encCipherEnigmaRotor4Position": "1",
        "encCipherEnigmaRotor3Position": "1",
        "encCipherEnigmaRotor2Position": "1",
        "encCipherEnigmaRotor1Position": "1",
        "encCipherEnigmaPlugboard": "",
        "encCipherEnigmaUhr": "0",
        "encCipherEnigmaUkwd": "",
        "encCipherJisKeyboardMode": "lenient",
        "encCipherScytaleKey": "2",
        "encCipherRailFenceKey": "2"
    }
}

How does the web know the default options? I don't want to hard code this in the plugin.

If I send a POST request myself with null options, the response is different than with the default options from the web request.

mozq commented 1 year ago

The issues with the .properties file have been fixed. Thank you for letting me know.

I've changed the default option values on the server side, could you try it with null? options parameter can be empty.

"options": {}
hlaueriksson commented 1 year ago

Thanks for fixing the .properties and the default options.

I noticed one diff when sending default options from the web vs empty options: image

mozq commented 1 year ago

I also changed the following default values in web. Please change these options. Sorry, I forgot to tell you that point. (DenCode saves option value to browser's localStorage. Please clear it if you want to check the default again.)

        "decCipherEnigmaRotor3": "III",  // "I" -> "III"
        "decCipherEnigmaRotor2": "II",
        "decCipherEnigmaRotor1": "I",  // "III" -> "I"

        "encCipherEnigmaRotor3": "III",  // "I" -> "III"
        "encCipherEnigmaRotor2": "II",
        "encCipherEnigmaRotor1": "I",  // "III" -> "I"
hlaueriksson commented 1 year ago

Yes, after clearing the local storage the response is identical. Thanks!

TheRamSan commented 7 months ago

Hi @mozq and @hlaueriksson I'm incredibly impressed with this project! The API usage thread has been super helpful, and I'm eager to start using it for decoding tasks like Quoted-printable code.

Could you please clarify a few details regarding API calls? I have some specific questions about encoding, body construction, and endpoints.

Here's a sample code I'd like to decode:

=FA=E5=E3=E4

Questions:

Encoding and Character Set:

Should I explicitly specify the encoding and character set when making a POST request? If so, should I use a query string or include it in the request body? Or does the API automatically detect these? Request Body Construction:

What's the correct way to format the request body for decoding tasks? Are there any specific examples or guidelines you can share? Endpoint:

Which specific endpoint should I use for decoding? Any additional information or documentation you can provide on API usage would be fantastic! I'm really keen to start exploring this tool's capabilities.

Thanks in advance for your help!

Truly appreciate your guidance. Ram

mozq commented 7 months ago

Hi @TheRamSan Thank you for your interest in this project. This project aims to perform encoding/decoding via a web UI. Therefore, the API endpoint is designed for the UI and is not intended to be called from outside the UI.

I’m not sure your usage purpose or environment, but since Quoted-printable has a simple specification, so I recommend implementing decoding logic within your application.

Please refer to this code for decoding implementation. https://github.com/mozq/dencode-web/blob/e6f892d84a84a87c6f77d370b2af0e729cc99a5d/src/main/java/com/dencode/logic/dencoder/StringQuotedPrintableDencoder.java#L167-L220

TheRamSan commented 7 months ago

Hello @mozq,

I appreciate your code and your fantastic website. The work you've done is impressive, and I'm truly grateful.

Currently, I'm using make.com to extract text from an eml file. Some of the text is encoded in Quoted-printable with the Windows-1255 character set, and I'm seeking a solution to decode it.

Unfortunately, within make.com, I'm restricted to running only vanilla JS code, and even that is not straightforward. Typically, if the required functionality is not available in make.com tools, we resort to making an HTTP request to an API that can provide the necessary information.

In this instance, I'm on the lookout for an API that can take a string as input and return the decoded text, preferably without the need to pre-select the character set, etc.

Do you happen to know of any such API?

Thanks again, Ram

mozq commented 7 months ago

Happy New Year, @TheRamSan !

Windows-1255 has only few characters defined, so I mapped the codes to Unicode manually. (I'm Japanese and I usually use a crazy amount of characters...) Please try the following vanilla JS code.

function decodeQP_windows1255(val) {
  const WINDOWS1255_MAP = {
    // See: https://en.wikipedia.org/wiki/Windows-1255
    0x80: '€', 0x82: '‚', 0x83: 'ƒ', 0x84: '„', 0x85: '…', 0x86: '†', 0x87: '‡', 0x88: 'ˆ', 0x89: '‰', 0x8B: '‹',
    0x91: '‘', 0x92: '’', 0x93: '“', 0x94: '”', 0x95: '•', 0x96: '–', 0x97: '—', 0x98: '˜', 0x99: '™', 0x9B: '›',
    0xA0: '\{A0}', 0xA1: '¡', 0xA2: '¢', 0xA3: '£', 0xA4: '₪', 0xA5: '¥', 0xA6: '¦', 0xA7: '§', 0xA8: '¨', 0xA9: '©', 0xAA: '×', 0xAB: '«', 0xAC: '¬', 0xAD: '\{AD}', 0xAE: '®', 0xAF: '¯',
    0xB0: '°', 0xB1: '±', 0xB2: '²', 0xB3: '³', 0xB4: '´', 0xB5: 'µ', 0xB6: '¶', 0xB7: '·', 0xB8: '¸', 0xB9: '¹', 0xBA: '÷', 0xBB: '»', 0xBC: '¼', 0xBD: '½', 0xBE: '¾', 0xBF: '¿',
    0xC0: '\u{5B0}', 0xC1: '\u{5B1}',  0xC2: '\u{5B2}',  0xC3: '\u{5B3}',  0xC4: '\u{5B4}',  0xC5: '\u{5B5}',  0xC6: '\u{5B6}',  0xC7: '\u{5B7}',  0xC8: '\u{5B8}',  0xC9: '\u{5B9}',  0xCA: '\u{5BA}',  0xCB: '\u{5BB}',  0xCC: '\u{5BC}',  0xCD: '\u{5BD}',  0xCE: '\u{5BE}',  0xCF: '\u{5BF}',
    0xD0: '\u{5C0}', 0xD1: '\u{5C1}',  0xD2: '\u{5C2}',  0xD3: '\u{5C3}',  0xD4: '\u{5F0}',  0xD5: '\u{5F1}',  0xD6: '\u{5F2}',  0xD7: '\u{5F3}',  0xD8: '\u{5F4}',
    0xE0: '\u{5D0}', 0xE1: '\u{5D1}',  0xE2: '\u{5D2}',  0xE3: '\u{5D3}',  0xE4: '\u{5D4}',  0xE5: '\u{5D5}',  0xE6: '\u{5D6}',  0xE7: '\u{5D7}',  0xE8: '\u{5D8}',  0xE9: '\u{5D9}',  0xEA: '\u{5DA}',  0xEB: '\u{5DB}',  0xEC: '\u{5DC}',  0xED: '\u{5DD}',  0xEE: '\u{5DE}',  0xEF: '\u{5DF}',
    0xF0: '\u{5E0}', 0xF1: '\u{5E1}',  0xF2: '\u{5E2}',  0xF3: '\u{5E3}',  0xF4: '\u{5E4}',  0xF5: '\u{5E5}',  0xF6: '\u{5E6}',  0xF7: '\u{5E7}',  0xF8: '\u{5E8}',  0xF9: '\u{5E9}',  0xFA: '\u{5EA}', 0xFD: '\u{200E}', 0xFE: '\u{200F}'
  };

  let s = '';

  for (let i = 0; i < val.length; i++) {
    const ch = val.charAt(i);

    if (ch == '=') {
      const nch = val.substring(i + 1, i + 3);
      i += 2;

      if (nch === '\r\n') {
        // Soft line break
        continue;
      }

      const cp = parseInt(nch, 16);

      if (cp <= 0x7F) {
        s += String.fromCharCode(cp);
      } else {
        s += WINDOWS1255_MAP[cp];
      }
    } else {
      s += ch;
    }
  }

  return s;
}
TheRamSan commented 7 months ago

Hi @mozq あけましておめでとうございます! It's been a long time since I was in japan - but I do remember the nice decorations for the new year in oodori park in sapporo.

Thanks for the code, I will give it a got. I really appreciate your help.

mozq commented 7 months ago

@TheRamSan Wow, ありがとうございます！どういたしまして！ I forgot 'u' in the mapping of 0xA0 and 0xAD. Please replace '\{}' to '\u{}'.

0xA0: '\u{A0}' 0xAD: '\u{AD}'

If you have any questions, please let me know.

P.S. It is more better to use an Array instead of an Object.

  const WINDOWS1255_MAP = [
    '€', '', '‚', 'ƒ', ......
  ];

  s += WINDOWS1255_MAP[cp - 0x80];
TheRamSan commented 7 months ago

const WINDOWS1255_MAP = [ '€', '', '‚', 'ƒ', ...... ];

s += WINDOWS1255_MAP[cp - 0x80];

Thank you again for your help. I really appreciate it. Ram

hlaueriksson commented 5 months ago

Hi again @mozq

I finally got around to release the DenCode plugin for PowerToys Run:

The X-Application-Id header is Community.PowerToys.Run.Plugin.DenCode:

mozq commented 5 months ago

Hi @hlaueriksson It is very easy to use and I love it! Please let me know If you need an API that returns a list of encoding methods and names, etc.

I've added a link to your project page in DenCode's README. Thank you.

hlaueriksson commented 5 months ago

I'm glad that you like the plugin. Thank you for an awesome site! The dark mode support was a nice surprise when I first saw it 😎

I got the encoding methods from the old config.properties file:

If I parse all the .java files in the dencoder folder in the master branch, can I get the same data?

I have a test that generates the encoding method data for the plugin. It's included in the binaries. I don't think you should add a API for this, if you don't plan to use it yourself.

mozq commented 5 months ago

I'm also glad you like dark mode😀

If I parse all the .java files in the dencoder folder in the master branch, can I get the same data?

Yes, you can parse the definitions from the .java files. However, definitions such as “useOe” are used for web UI, so you do not need to get them. It's easier to parse “*.method=” in messages.properties. The file also includes names and descriptions.

Please note that the key such as “label.encStrBin=” will be changed to “string.bin.encStrBin=”, but I'm too lazy to do it yet :P Once the key changes, it becomes more easier to parse.

mozq commented 5 months ago

I’ve changed key name of dencoder functions in messages.properties as below.

string.bin.method=Bin String
string.bin.title=……
string.bin.desc=……
string.bin.tooltip=……
string.bin.func.encStrBin=Bin String
string.bin.func.decStrBin=Bin String

If dencoder methods or functions are added, renamed, or deleted in the future, please parse this file and synchronize it.

hlaueriksson commented 5 months ago

Thanks! I will take a look at messages.properties and update my parsing code.

mozq commented 5 months ago

Hi @hlaueriksson FYI, here is a sample code for messages.properties parser. (This code does not support "string.naming-convention" method. Please ignore the method.)

using System.Text.Json;
using System.Text.RegularExpressions;

Regex keyRegex = new("^(?<method>[^\\.]+\\.[^\\.]+)\\.(?<subkey>method|title|desc|toltip|func)(?:\\.(?<function>.+))?");

Dictionary<string, Dictionary<string, string>> methods = [];
Dictionary<string, Dictionary<string, string>> methodFunctions = [];

string messages = await new HttpClient().GetStringAsync("https://raw.githubusercontent.com/mozq/dencode-web/master/src/main/resources/messages.properties");

StringReader reader = new(messages);
string? line;
while ((line = reader.ReadLine()) != null)
{
    int idx = line.IndexOf('=');
    if (idx == -1)
    {
        continue;
    }

    string key = line[..idx];
    string label = line[(idx + 1)..].Replace("\\\\", "\\");

    Match m = keyRegex.Match(key);
    if (!m.Success)
    {
        continue;
    }

    string method = m.Groups["method"].Value;
    string subkey = m.Groups["subkey"].Value;
    string function = m.Groups["function"].Value;

    if (subkey == "method")
    {
        methods[method] = [];
        methods[method]["key"] = method;
        methods[method][subkey] = label;

        methodFunctions[method] = [];
    }
    else if (subkey == "func")
    {
        methodFunctions[method][function] = label;
        //methodFunctions[method[..method.IndexOf('.')] + ".all"][function] = label;
        //methodFunctions["all.all"][function] = label;
    }
    else
    {
        methods[method][subkey] = label;
    }

}

Console.WriteLine(JsonSerializer.Serialize(methods));
Console.WriteLine(JsonSerializer.Serialize(methodFunctions));
hlaueriksson commented 5 months ago

Thanks for the sample code @mozq

I continued with my naive approach without regex:

mozq commented 5 months ago

Thank you for implementing it.