Version 3

[m-vo]

Closes #38, #42, #44, #45, #47, #48

This PR removes a lot of old dependencies and brings the following changes:

• Some minimum requirements needed to be bumped, most notably PHP to 8.1 and Contao to 4.13.

• The import now uses the Symfony HTTP Client and makes use of streaming responses asynchronously. This drastically improves the import speed (<30s for me for a full sync).

• There are two new CLI parameters: timeout and connections. By increasing the number of connections, more parallel requests will be made. E.g. immoscout24:sync --connections=50 will use up to 50 parallel requests. If needed the default timeout (per connection) can also be increased. Currently we are failing hard if any connection times out or other network related issues appear in order to notice that something's going wrong (without modifying data). This happened several times to me when trying - especially for high connection numbers (the API basically stopped responding) but everything went fine again a few seconds later. So no big deal.

• Attachments are now automatically scraped. No need for the second command anymore. Also, this uses asynchronous requests to speed thing up.

• Attachments are stored in their own Virtual Filesystem (Contao 4.13 feature). By default this is mounted to /immoscout24. When a real estate object gets removed from the database (when syncing), the respective files get deleted as well. There is no DBAFS anymore - the files are now fully managed by the extension.

• I removed all the hacky helper methods from the Attachment to render an image, and instead added a template helper function getFigureFromAttachment. This allows generating a Contao Figure object from an attachment, that can be queried for its data or used to render an arbitrary template.

By implementing and refactoring the above, I also fixed several bugs (see linked issues above).

Due to the breaking changes (API wise), this must be released as a new major version (2.0).

[m-vo]

/cc @Metis77

[Metis77]

while testing I am always running into memory size issues. event after setting the php memory_limit = 10000M.

any idea?

[Metis77]

Update: this worked, but that should not be the solution (?) php -d memory_limit=-1 vendor/bin/contao-console immoscout24:sync

[Metis77]

Beim hinzufügen eines Templates bekomme ich den Fehler:

[2022-11-05T14:38:49.171952+00:00] request.INFO: Matched route "contao_backend". {"route":"contao_backend","route_parameters":{"_route":"contao_backend","_scope":"backend","_token_check":true,"_controller":"Contao\\CoreBundle\\Controller\\BackendController::mainAction"},"request_uri":"https://local.is24.derhaeuptling.de/contao?do=tpl_editor&key=new_tpl&ref=...","method":"POST"} []
[2022-11-05T14:38:49.195462+00:00] request.CRITICAL: Uncaught PHP Exception RuntimeException: "Invalid path ../bundles/contao-immobilienscout24-import-bundle/src/Resources/contao/templates/mod_immoscout24_real_estate_list_teaser.html5" at /var/cache/prod/contao/dca/tl_templates.php line 114 {"exception":"[object] (RuntimeException(code: 0): Invalid path ../bundles/contao-immobilienscout24-import-bundle/src/Resources/contao/templates/mod_immoscout24_real_estate_list_teaser.html5 at /var/cache/prod/contao/dca/tl_templates.php:114)"} []

[m-vo]

event after setting the php memory_limit = 10000M.

Your screenshot shows "134217728 bytes" - that's only 128MiB, though, and will probably be too little to import such a huge amount of data. I can try optimizing, but I'm not sure this is possible.

[Metis77]

Stack Trace :

Invalid path ../bundles/contao-immobilienscout24-import-bundle/src/Resources/contao/templates/mod_immoscout24_real_estate_list.html5

RuntimeException:
Invalid path ../bundles/contao-immobilienscout24-import-bundle/src/Resources/contao/templates/mod_immoscout24_real_estate_list.html5

  at vendor/contao/core-bundle/src/Resources/contao/dca/tl_templates.php:292
  at tl_templates->addNewTemplate(object(DC_Folder))
     (vendor/contao/core-bundle/src/Resources/contao/classes/Backend.php:446)
  at Contao\Backend->getBackendModule('tpl_editor', null)
     (vendor/contao/core-bundle/src/Resources/contao/controllers/BackendMain.php:168)
  at Contao\BackendMain->run()
     (vendor/contao/core-bundle/src/Controller/BackendController.php:49)
  at Contao\CoreBundle\Controller\BackendController->mainAction()
     (vendor/symfony/http-kernel/HttpKernel.php:163)
  at Symfony\Component\HttpKernel\HttpKernel->handleRaw(object(Request), 1)
     (vendor/symfony/http-kernel/HttpKernel.php:75)
  at Symfony\Component\HttpKernel\HttpKernel->handle(object(Request), 1, true)
     (vendor/symfony/http-kernel/Kernel.php:202)
  at Symfony\Component\HttpKernel\Kernel->handle(object(Request))
     (public/index.php:44)                

[m-vo]

We'll handle the Contao issue over at the Contao repository. :slightly_smiling_face:.