Add Diego Del Llano's content

[sebamarynissen]

This PR adds a big part of Diego Del Llanos content. Most of the content was added by using a parsing script that generates the yaml automatically.

Still a work in progress because it needs some testing, so don't merge yet. I just created the PR already so that no-one would be doing double work.

Certain content could not be added because it uses a .rar format for some reason:

• https://community.simtropolis.com/files/file/31556-torre-mayor/
• https://community.simtropolis.com/files/file/33091-figueroa-at-wilshire/
• https://community.simtropolis.com/files/file/32279-republic-plaza/

Might be added later if sc4pac supports .rar as well.

[sebamarynissen]

@memo33 I'm trying to get the GitHub action to pass, but it chokes on some files that only have a dark night variant. What should happen in these cases? I now made them look like

group: diego-del-llano
name: citigroup-center-new-york
variants:
  - variant: { nightmode: dark }
    dependencies: ["simfox:day-and-nite-mod"]
    assets:
      - assetId: diego-del-llano-citigroup-center-new-york-darknite

so I suppose I just get rid of the variants field, but still add simfox:day-and-nite-mod as a dependency?

group: diego-del-llano
name: citigroup-center-new-york
dependencies: ["simfox:day-and-nite-mod"]
assets:
  - assetId: diego-del-llano-citigroup-center-new-york-darknite

I'm curious though what would happen in that case if someone does not have darknite installed, and then tries to install sc4pac diego-del-llano:citigroup-center-new-york. I suppose it will install simfox:day-and-nite-mod. Will that automatically update then packages which do have a darknite version?

[sebamarynissen]

I think I've finished the entire collection. Tested both darknite and maxisnite, and all files seem to be properly present, though it's possible that I missed something given the sheer amount of files (more than 100).

For future reference, I've posted the Node.js script that I used to generate most of the files below. Could be useful to reuse it for other creators as well perhaps.

import fs from 'node:fs';
import { JSDOM } from 'jsdom';

async function generate(input) {

  let url = new URL(input);
  url.searchParams.delete('confirm');
  url.searchParams.delete('t');
  url.searchParams.delete('csrfKey');
  let offset = +(url.searchParams.get('offset') || 1);
  url.searchParams.delete('offset');

  let next = new URL(url);
  next.searchParams.set('r', +url.searchParams.get('r')+offset);

  let link = new URL(url+'');
  for (let key of [...link.searchParams.keys()]) {
    link.searchParams.delete(key);
  }
  link = String(link);

  let html = await fetch(link).then(res => res.text());
  let jsdom = new JSDOM(html);
  let h2 = [...jsdom.window.document.querySelectorAll('h2')].find(node => node.textContent.toLowerCase().includes('about this file'));
  let text = [...h2.nextElementSibling.querySelectorAll('div > p')]
    .filter(p => {
      if (p.textContent.includes('notice')) return false;
      if (p.textContent.includes('File content')) return false;
      return true;
    })
    .slice(0, 2)
    .map(p => p.textContent.trim())
    .map(p => p.replaceAll('\u00a0', ' '))
    .filter(Boolean)
    .map(p => {
      return '    ' + p.split('. ').join('.\n    ');
    })
    .join('\n\n');

  let updated = [...jsdom.window.document.querySelectorAll('time')].find(time => {
    return time.parentElement?.previousElementSibling?.textContent.includes('Updated');
  }) || [...jsdom.window.document.querySelectorAll('time')].find(time => {
    return time.parentElement?.previousElementSibling?.textContent.includes('Submitted');
  });
  let modified = updated.getAttribute('datetime');

  let path = link.replace(/\/$/, '').split('/').at(-1);
  const regex = /[\d+]-(.*)/;
  let [, id] = path.match(regex);

  let hasMaxisNight = !!url.searchParams.get('r');

  let output = `group: diego-del-llano
name: ${id}
version: 1.0.0
subfolder: 300-commercial
info:
  summary: ${id.split('-').map(word => word.charAt(0).toUpperCase() + word.slice(1)).join(' ')}
  description: >
${text}

  author: Diego Del Llano
  website: ${link}

variants:
  - variant: { nightmode: dark }
    dependencies: ["simfox:day-and-nite-mod"]
    assets:
      - assetId: diego-del-llano-${id}-darknite
`;
  if (hasMaxisNight) output += `  - variant: { nightmode: standard }
    assets:
      - assetId: diego-del-llano-${id}-maxisnite
`;

  output += `
---
assetId: diego-del-llano-${id}-darknite
version: 1.0.0
lastModified: "${modified}"
url: ${url}
`;
  if (hasMaxisNight) output += `
---
assetId: diego-del-llano-${id}-maxisnite
version: 1.0.0
lastModified: "${modified}"
url: ${next}
`;

  fs.writeFileSync(`src/yaml/diego-del-llano/${id}.yaml`, output);

}

let urls = `
// Paste download urls here
`;

for (let url of urls.trim().split('\n')) {
  await generate(url.trim());
}

[memo33]

Awesome, thanks a lot. I'm going to need some time for the review.

Certain content could not be added because it uses a .rar format for some reason:

I'm about to add support for .rar files.

I'm curious though what would happen in that case if someone does not have darknite installed, and then tries to install sc4pac diego-del-llano:citigroup-center-new-york. I suppose it will install simfox:day-and-nite-mod. Will that automatically update then packages which do have a darknite version?

Incomplete variants are not allowed as they could lead to unresolvable conflicts (for example, if you've previously chosen MN and then install such a DN-only package as dependency), so all possible variants must be included in the yaml files. In this case there are two options:

• (a) don't add variants, but simply depend on simfox:day-and-nite-mod (like you suggested),
• (b) include both MN and DN variants, but install the same DN models in either case, and only depend on simfox:day-and-nite-mod in the DN case.

The end result is almost the same practically, so I don't have a strong opinion yet on what's preferable, but for semantic reasons I'm leaning towards (b).

The dark night mod is configured to install just an empty folder when choosing nightmode: standard https://github.com/memo33/sc4pac/blob/6f1457302cbcc3fc92f0f5f5e53d12bfb65615fa/src/yaml/simfox/day-and-nite-mod.yaml#L6-L10 so it's safe to depend on it even in case (a). This would not force all previously installed packages to switch to the DN variant. It's just an unnecessary dependency for MN users.

[sebamarynissen]

Cool, thanks for the information. I've gone with option (a) in 353b2bbb2d0868ed3a0b458bced8fe3beb67dbce, but I you want I can change the files again to option (b). I do prefer this too actually.

[memo33]

Ok, let's go with (b) then.

[memo33]

Are all of these ploppable landmarks? I think we should make this clearer in the documentation, at least by adding a new subfolder such as 360-landmark or similar. So far, there aren't any landmarks yet.

Some of the descriptions could probably be shortened a bit to include just the more essential infos, but I understand that this is easier said than done.

The version numbers don't seem to match the original uploads. It's debatable whether there's a difference between 1.0.0 and 1.0, but maybe there's a way to fix this in the script.

[sebamarynissen]

I think they are all landmarks, allthough there could be some growables. I don't recall having seen growables anywhere though.

For the descriptions, I just configured my script to take the first two paragraphs of what was entered on Simtropolis (which is often just text from Wikipedia actually). Not sure what the description should be if we change it to something else.

About the version numbers, ugh, I did not notice that they could've been parsed from Simtropolis as well. The thing is that I can't really run the generate script again because even though it did most of the work, a lot of manual interventions were still needed. I guess I could write a new script that parses just the version informations though.

[sebamarynissen]

I've pushed a few additional commits where I:

• Switched to option (b) for the darknite-only packages
• Changed the version numbers with a script so that they automatically match the version numbers on Simtropolis

For future reference, the script I used was this:

import fs from 'node:fs';
import path from 'node:path';
import yaml from 'yaml';
import { JSDOM } from 'jsdom';

let dir = path.join(import.meta.dirname, './src/yaml/diego-del-llano');
let tasks = fs.readdirSync(dir).map(async filename => {
  let fullPath = path.join(dir, filename);
  let buffer = fs.readFileSync(fullPath).toString('utf8');
  let parsed = yaml.parse(buffer.split('---')[0]);
  let { website } = parsed.info;
  if (!website.endsWith('/')) return;

  let html = await fetch(website).then(res => res.text());
  let jsdom = new JSDOM(html);
  let version = jsdom.window.document.querySelector('.stex-title-version').textContent;
  if (version !== parsed.version) {
    console.log('changing '+website);
    buffer = buffer.replaceAll(/version: [\d\.]+/g, `version: "${version}"`);
    console.log(buffer);
    fs.writeFileSync(fullPath, buffer);
  }
});
await Promise.all(tasks);

[sebamarynissen]

@memo33 I've added the 3 missing packages that used the .rar format and tested it with your updated version of sc4pac. Looks like it's working properly.

Sidenote: this means that this PR can't be merged of course before a new version of sc4pac with support for rar archives is officially released.

[memo33]

Thanks. I've made a few more tweaks.

[memo33]

Capella Tower seems to include the same building twice. Is that a bug? Should we exclude one of them?

[sebamarynissen]

Looks like the Capella Tower actually consists out of two buildings, so seems okay.

[memo33]

Well ok, with nightmode: standard there seem to be 3 buildings, two of which are the tower. This duplication doesn't exist with nightmode: dark.

[sebamarynissen]

Ah okay, I didn't notice that. Good catch!