duri0214 / portfolio

0 stars 1 forks source link

sec: VPSだとなぜか arelle の処理が終わらない #81

Open duri0214 opened 2 months ago

duri0214 commented 2 months ago

https://groups.google.com/g/arelle-users/c/2a0oXtI5qA4

class XbrlService:
       :
    def make_counting_data(self, work_dir: Path) -> CountingData:
        print(11)
        temp_dir = Path(work_dir) / "temp"
        if not temp_dir.exists():
            temp_dir.mkdir(parents=True, exist_ok=True)
        print(12)

        ZipFileService.extract_zip_files(work_dir, temp_dir)
        xbrl_path = str(next(temp_dir.glob("XBRL/PublicDoc/*.xbrl")))
        print(13)
        ctrl = Cntlr.Cntlr()
        print(131)
        model_xbrl = ctrl.modelManager.load(xbrl_path)
        print(14)
        [logging.info](http://logging.info/)(f"  xbrl: {Path(xbrl_path).name}")
        print(15)
        counting_data = self._assign_attributes(
            counting_data=CountingData(), facts=model_xbrl.facts
        )
        print(16)
        shutil.rmtree(temp_dir)
        print(17)

        return counting_data

131 の部分 ctrl.modelManager.load で処理が止まってしまう

(venv) ubuntu:/var/www/html/portfolio$ python manage.py daily_download_edinet
S100QKJ2 download...
S100QKJ2 download...done
1
11
12
13
131
[Errno -3] Temporary failure in name resolution
retrieving http://disclosure.edinet-fsa.go.jp/taxonomy/jpdei/2013-08-31/label/jpdei_2013-08-31_lab-en.xml
Forced termination

メモリを1GB以上使ってんのが気になるんだよな

[255408.493315] Out of memory: Killed process 725 (mysqld) total-vm:1822024KB,
anon-rss:418228KB, file-rss:3072KB, shmem-rss:0KB, UID:108 pgtables:1400KB oom_score_adj:0
[255426.608838] Out of memory: Killed process 40512 (mysqld) total-vm:1786156KB,
anon-rss:360676KB, file-rss:2816KB, shmem-rss:0KB, UID:108 pgtables:1240KB oom_score_adj:0
[256443.240592] Out of memory: Killed process 40570 (mysqld) total-vm:1038164KB,
anon-rss:334928KB, file-rss:2816KB, shmem-rss:0KB, UID:108 pgtables:1076KB oom_score_adj:0

ping もタイムアウトする

PS C:\Users\yoshi> ping [disclosure.edinet-fsa.go.jp](http://disclosure.edinet-fsa.go.jp/)

Pinging [s-part-0018.t-0009.t-msedge.net](http://s-part-0018.t-0009.t-msedge.net/) [2620:1ec:bdf::46] 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 2620:1ec:bdf::46:
    Number of packets: sent = 4, received = 0, loss = 4 (100% loss),
PS C:\Users\yoshi>
duri0214 commented 2 months ago

メモリ使い切ってる? さくらVPSのメモリ状況を見れればなぁ